Gl_NormalMatrix replacement in shaders

First let me apologize for asking what I am sure is an overly asked question but all my searching and testing has resulted in nothing.

I am attempting to remove all the GLSL built-ins from my shaders and the last one for #version 150 is gl_NormalMatrix. From all my reading it’s the transposed inverse of the upper 3x3 of the product of the Model and View matrices. So I have tried numerous permutations of this line (mat3 glNormalMatrix = transpose(inverse(mat3(mView * mModel)))) with no success replicating gl_NormalMatrix.

Below is a normal mapping vertex shader and it works perfectly with the gl_NormalMatrix built in but this built in is depreciated in version > 120 thus the exercise.

I am making some assumptions here but I am pretty certain may matrices are correct and I pass a projection, view(camera) and model(world) matrix in to transform the object coordinates. Additionally I have annotated what space I believe the vec3 to be in and what direction the mat4 is intended to transform to. I don’t think it’s a problem with my TBN but not ruling that out also. I just assumed I’d be able to calculate a ‘replacement’ for gl_NormalMatrix from the components passed in but that is proving to be harder than I had suspected. The offending line is in the CalculateTBN near the bottom with the working line uncommented and the ‘what I think is correct’ commented.

I don’t rule out the TBN calculations but I believe they are correct as they work with the gl_NormalMatrix.

If seeing the fragment shader or my TBN calculations will help I will post but I suspect those components may be a side matter to this activity. Also this shader is not optimized I know but I’m following the mantra, make it work (built-ins), make it work right (remove built-ins), make it fast (move some or most of the matrix math back to CPU). Maybe that exercise will solve this problem but I suspect I should be able to replicate a replacement for gl_NormalMatrix in the shader also. Additionally any insight into making it better is always appreciated.

Notes on what’s happening when rendering a plant with normal maps. With gl_NormalMatrix the lighting seems correct and stable to light source. Normal map looks correct, specular stay with camera angle. With my attempts to replicate the gl_NormalMatrix the dark side rotates with the globe, specular does odd stuff like going opposite directions, culminating at points on the globe and other odd effect. The normal mapping seems to be correct as it still looks accurate but with the shaded and specular moving it can be hard to tell.

Any insight is greatly appreciated…

//#version 150
uniform mat4 mProjection;                                                         // to clip space  (projection)
uniform mat4 mView;                                                               // to view space  (camera)
uniform mat4 mModel;                                                              // to world space (world)
uniform vec3 vLightSourcePosition;                                                // world space
uniform vec3 vCameraSourcePosition;                                               // world space
attribute vec3 a_vVertex;                                                         // incoming vertex (object space)
attribute vec3 a_vNormal;                                                         // incoming normal (object space)
attribute vec2 a_vTexel;                                                          // incoming normal (object space)
attribute vec3 a_vTangent;                                                        // to tangent space
/* ------------------------------------------------------------------------------------------ */
varying vec2 vTexCoord;
varying vec3 vLightDirection;                                                     // object space
varying vec3 vCameraDirection;                                                    // object space
mat3 CalculateTBN(vec3 vNormal, vec3 a_vTangent);
void main(void)
  vec3 vVertex = a_vVertex;                                                       // object space
  vec3 vNormal = a_vNormal;                                                       // object space
  vTexCoord = a_vTexel;
  // Calculate the positions from the model using the inverse of the model 
  //  Move from model space to local (object space)
  vec3 vLightPosition = vec3(inverse(mModel) * vec4(vLightSourcePosition, 1));    // object space
  vec3 vCameraPosition = vec3(inverse(mModel) * vec4(vCameraSourcePosition, 1));  // object space
  mat3 TBN = CalculateTBN(vNormal, a_vTangent);
  vLightDirection = TBN * (vLightPosition - vVertex);                             // tangent space
  vCameraDirection = TBN * (vCameraPosition - vVertex);                           // tangent space
  // Calculate the vertex position by combining the model/view(camera)/projection matrices
  gl_Position = mProjection * mView * mModel * vec4(a_vVertex, 1);                // clip space
mat3 CalculateTBN(vec3 vNormal, vec3 a_vTangent)
  mat3 glNormalMatrix = gl_NormalMatrix;                                          // to normal space??
  //mat3 glNormalMatrix = transpose(inverse(mat3(mView * mModel)));               // to normal space??
  // Calculate the Inverse TBN  (to transform from object space to tangent space)
  vec3 n = normalize(glNormalMatrix * vNormal);
  vec3 t = normalize(glNormalMatrix * a_vTangent);
  vec3 b = cross(n, t);
  mat3 TBN = transpose(mat3(t,b,n));                                              // to tangent space
  return TBN;

If you are only doing uniform scaling, AFAIK mat3(nView*nModel) will work; but it is more normal to calculate it on the cpu and pass it as another unform. Also I would not be inclined to use variables names of the form “gl_xx” for my own variabes.

I will say I am thrown by the term ‘uniform scaling’. What are you referring? The mView is the camera’s matrix and the mModel is the model’s matrix. Both are, on the CPU side, 4x4 matrices for rotation, scaling(of which i am not scaling) and translations. Camera matrix is all negative values of position/orientation and multiplication is reversed but seem to be compatible with the gl_Position calculations and the inverse(mModel) calls to transform camera an light ‘world’ positions into object space.

mView = vect2transmat(-pos) * quat2mat(-qOri);
mModel = quat2mat(qOri) * vect2transmat(pos);

When it comes to uniform scaling, I have also seen reference to orthogonal rotation matrices which I assume is X = Y = Z type of rotations which is not what I have here. I am actually building the required matrix from a Quaternion for orientation and a Vector3 for position.

As far as the gl_XX variables names I agree. I assume your referencing the glNormalMatrix variable in the CalculateTBN call. I used that name as a quick replacement for the gl_NormalMatrix just to be lazy in the code edits. My plan is to optimize this my removing all the unneeded calculations like the inverse(mModel) and (if I can figure it out) the glNormalMatrix to the CPU side. I may even go as far as passing in both a_vTangent and BiNormal(BiTangent). BTW…the a_ is for attributes, and thinking about moving to u_ and v_ for uniform and varying respectively but it appears that those may have also been depreciated to in/out).

So…I tried as you suggested, and as I believe I have, and the results are similar to before. The lighting seems to be bound to the model and camera position seems to have some influence…

EDIT…mat3 glNormalMatrix = mat3(mView * mModel); // to normal space??

Could something be transposed that I am not aware of and it getting washed out in the calculations? I’d suspect no but I am kind of blind when it comes to the inner workings of shaders.

Thanks for the response…this is a tough one for me…any thoughts on how to extract the gl_NormalMatrix values from the shader for examination. If I could see it I should be able to reverse it and then replicate it.

OK…so more research on orthogonality has delivered this. My upper left 3x3 of my ModelView matrix is orthogonal. All rows and columns have a length of one or as close as it will get. The transpose of the ModelView equals the Inverse of the ModelView so I think I have answered the question of uniform scaling if indeed that is what you were getting at.

So…Is it my TBN? I cannot say. If works with the GL supplied normal matrix. All I can assume is there is some piece of magic I am missing. If I could extract the gl_NormalMatrix values from the shader I could attempt to reverse this and get an idea of whats wrong.

From all accounts I am calculating the normal matrix properly…

Any thoughts?

gl_NormalMatrix = transpose(inverse(modelview)); //this link explains it all, if you are interested in mathematical reasons for transpose and inverse

google query “glsl normal matrix”;

OK…here is a puzzle.

In this case gl_NormalMatrix == mat3(1);


I did some testing with vDebugColor = gl_NormalMatrix[x][y] and passed through to fragment shader and multiplied to final color on output. Went through all 9 iterations and 0,0; 1,1; 2,2 = 1 other = 0. Where it was 1 I saw my model rendered in perfect color, where it was 0 the model was black, rendered but black.

So I set my variable glNormalMatrix to mat3(1) and it worked. Took gl_NormalMatrix out all together and it worked.

mat3 CalculateTBN(vec3 vNormal, vec3 a_vTangent)
// Calculate the Inverse TBN (to transform from object space to tangent space)
vec3 n = [b]normalize/b;
vec3 t = [b]normalize/b;
vec3 b = cross(n, t);
mat3 TBN = transpose(mat3(t,b,n));

// to tangent space      
return TBN; 


So now the question. Whats up? Why would gl_NormalMatrix be mat3(1)? Also if that’s the case then I believe it’s telling me my tangents are in object space not tangent space. I suspect this as my normals are an attribute and in object space. The matrix created uses normal(attribute), tangent(attribute) and a cross(assume in object also). Thus my calculations to convert to tangent space are wrong.

Anyone ever seen gl_NormalMatrix = mat3(1)? Any reason this may be the case?

BTW…I’ve read the lighthouse tutorial and many others backwards and forwards. I never suspected an identity matrix.

I would love to hear someones explanation of why gl_NormalMatrix could be identity.

Its late…gotta sleep on this one.

gl_NormalMatrix doesn’t know what a tangent or bitangent is. That’s not what it’s for, so it being identity means nothing about your tangents or whatever. It’s just the inverse/transpose of the upper 3x3 portion of the modelview matrix. That’s all.

If it’s identity, then odds are good that gl_ModelViewMatrix is also identity.

One more thought…I am calculating a mat3 and transforming by that the (pos - vertex). I’ll need to see if mat3 transform is OK vs. 3 dot products. One for each of the x, y, z components.

tvec = lpos - vertex;
lvec.x = dot(tvec, t);
lvec.y = dot(tvec, b);
lvec.z = dot(tvec, n);


lvec = tbn * (lpos - vert);

Ultimatly I think I have things in the wrong spaces, (clip,view,world,object,tangent).

vCameraDirection = TBN * (vCameraPosition - vVertex);

try replacing it with

vCameraDirection = TBN * (-vVertex);

Wait a second…since this is all shader based and I am not using glMatrix…() then your correct, everything is identity.

I am attempting to emulate OpenGL ES on the desktop. So I set GL_PROJECTION and GL_MODELVIEW to identity so the fixed pipeline doesn’t get in the way. Since I want to eventually move to a mobile platform, I know there will be other shader challenges there, I wanted to attempt to get the fixed pipeline out of the way. Since there appears to be no ES mode for the desktop GL I figured identities would cause nothing to happen. That’s probably whats happening.

How would one get the desktop GL to stop using the fixed functionality? This is a quick development setup so I am using C# and OpenTK with TAO compatibility for easy port back to C++ later. The only real OpenTK I am using are the Math libraries and the GL control for windows forms. Everything else I am trying to keep pure C# and only the GL calls required like glActiveTexture, glSetUniform, etc… and the like.

I never thought to consider the identities for projection and modelview I set at the start of the program. It was just a way to get the fixed functionality to be quiet.

i don’t see any feedback about my last suggestion.

No feedback as I have not tried yet. Sleep and work intervene. Although, after a few hours of reflection I think I understand what was happening as stated in my last response. Setting both projection and modelview to identity causes the gl_NormalMatrix to be identity. Makes sense. I think I ran into this trying to emulate OpenGLES on a desktop platform. Start started small and started moving to pure shader world with the side effects of the Desktop GL ‘helping’.

That being said I also think my tangent and bi-tangent are incorrect as far as being in tangent space. I believe now, and always suspected, they are in object space. The calculation takes into account vertex and texture directions and makes some minor calculations. I believe the results are in object space. I have an addition piece I am not calling to calculate the inverse which should move to tangent space. With that I’m not sure what I’ll gain by doing this. I’d then have to move the light and camera vectors into tangent space. Calculation already being preformed but for object space. Something I need to think about since normals on the map are in tangent space something has to give.

So ultimately I was calculating the normal matrix properly, the shaders were getting an identity so whatever I attempted to replace with made things worse. I think this specific shader requires some thought and rework but as it appears to be working I will work with it, unless someone see off-hand where the object/tangent space resides. It may be in the fragment shader since that’s where the normal is actually extracted.

Seems like I went from a ‘how to compute normal matrix’ to ‘how to better emulate OpenGL ES and its lack of the fixed pipeline’.

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.