Matrix Inverse & Broken Vertex Shader

Hey everone. In my vertex shader, I reckon I’m not calculating the inverse() of a mat3 properly. I am trying to get the brick shader demo from the orange book to work and I’ve got it to render, but the lighting looks a little off. I would have used the built in inverse() that the book claims exists for >= 1.40 (table 5.5 of the Orange Book (p137)), but my nvidia driver complains it doesn’t have it:

warning C7547: extension GL_EXT_gpu_shader5 not supported in profile gp4vp

What is a shader profile anyways and how does one select one in the first place?

If I remove the #extension, I get…

error C7531: global function inverse requires "#extension GL_EXT_gpu_shader5 : enable" before use

Since inverse() doesn’t seem to be implemented, I’ve tried implementing it myself. The code for my vertex shader is as follows. I am confident the fragment shader is ok.

  Name:         brick.glslv
  Description:  Vertex shader to procedurally generate brick texture on model...

// We want at least GLSL 1.50...
#version 150

// We need this for inverse(), but I thought it was already supported in 
//  GLSL 1.40...
#extension GL_EXT_gpu_shader5 : enable

// Vertex attributes...

    // Vertex position in model coordinates...
    in vec4 mcVertex;

    // Normal plane / vector in model coordinates. Assume already normalized...
    in vec3 mcNormal;

// Uniform variables...

    // Transformation matrices...
    uniform mat4 ModelViewMatrix;
    uniform mat4 ProjectionMatrix;
    // Light position in eye coordinates...
    uniform vec3 ecLightPosition;

// Constants...

    // Lighting parameters...
    const float SpecularContribution    = 0.3;
    const float DiffuseContribution     = 1.0 - SpecularContribution;

// Outputs to be interpolated for the fragment shader...

    // Intensity of light at this vertex...
    out float LightIntensity;
    // Position of vertex to be interpolated in model coordinates...
    out vec2  mcPosition;

// Calculate inverse, in case built-in not supported...
mat3 inverse(const in mat3 Matrix)
    // Calculate the determinant...
    float D = determinant(Matrix);
        // Singular matrix, problem...
        if(D == 0.0)
            return mat3(0.0);

    // Calculate the transpose...
    mat3 MatrixT = transpose(Matrix);

    // Calculate needed determinants...
    float D00 = MatrixT[1][1] * MatrixT[2][2] + MatrixT[2][1] * MatrixT[1][2];
    float D10 = MatrixT[0][1] * MatrixT[2][2] + MatrixT[2][1] * MatrixT[0][2];
    float D20 = MatrixT[0][1] * MatrixT[1][2] + MatrixT[1][1] * MatrixT[0][2];
    float D01 = MatrixT[1][0] * MatrixT[2][2] + MatrixT[2][0] * MatrixT[1][2];
    float D11 = MatrixT[0][0] * MatrixT[2][2] + MatrixT[2][0] * MatrixT[0][2];
    float D21 = MatrixT[0][0] * MatrixT[1][2] + MatrixT[1][0] * MatrixT[0][2];
    float D02 = MatrixT[1][0] * MatrixT[2][1] + MatrixT[2][0] * MatrixT[1][1];
    float D12 = MatrixT[0][0] * MatrixT[2][1] + MatrixT[2][0] * MatrixT[0][1];
    float D22 = MatrixT[0][0] * MatrixT[1][1] + MatrixT[1][0] * MatrixT[0][1];
    // Assemble matrix of cofactors...
    mat3 MatrixAdjugate;
    MatrixAdjugate[0] = vec3( D00, -D01,  D02);
    MatrixAdjugate[1] = vec3(-D10,  D11, -D12);
    MatrixAdjugate[2] = vec3( D20, -D21,  D22);
    // Calculate the inverse...
    return (1.0 / D) * MatrixAdjugate;

// Entry point...
void main()
    // Variables...
    float   Specular;
    mat4    ModelViewProjectionMatrix   = ProjectionMatrix * ModelViewMatrix;
    mat3    NormalMatrix                = inverse(mat3(transpose(ModelViewMatrix)));

    // Calculate the vertex position in eye coordinates...
    vec3 ecPosition = vec3(ModelViewMatrix * mcVertex);
    // Calculate the normalized vector to the light source from vertex...
    vec3 ecLightVector = normalize(ecLightPosition - ecPosition);
    // Calculate the normal vector in eye coordinates...
    vec3 ecNormalVector = normalize(NormalMatrix * mcNormal);
    // Calculate vector to viewer from vertex (viewer is origin in eye coordinates)
    vec3 ecViewerVector = normalize(-ecPosition);

    // Given the vector from surface to light and normal, compute reflected ray...
    vec3 ecReflectionVector = reflect(-ecLightVector, ecNormalVector);

    // Diffuse light at this vertex is a function of how coincident normal and
    //  the incident light wave are...
    float Diffuse = max(dot(ecLightVector, ecNormalVector), 0.0);

    // Specular light at this vertex is a function of how coincident normal and
    //  viewer are, but don't waste time if surface doesn't even face the light...
    if(Diffuse > 0.0)
        // Calculate...
        Specular = max(dot(ecReflectionVector, ecViewerVector), 0.0);
        // Amplify...
        Specular = pow(Specular, 16.0);
        Specular = 0.0;

    // Output light intensity to be interpolated for fragment shader...
    LightIntensity = (Diffuse * DiffuseContribution) + 
                     (Specular * SpecularContribution);
    // Output vertex position in clipping coordinates...
    gl_Position = ModelViewProjectionMatrix * mcVertex;

    // Output vertex position to be interpolated for fragment shader...
    mcPosition = mcVertex.xy;

Any help appreciated.



  1. The inverse() function is only available for GLSL >= 1.50 :

  2. Why do you have an addition in your determinant calculation and no substraction?

  3. It is not very efficient to calculate the inverse in the shader, because the matrix is the same for every vertex. You should do this in application.

  4. I dont know if it makes a difference, but actually the normal matrix is the transposed inversed modelview submatrix ( not the inversed transposed ) :


Hey Dj3hut. Thanks for your response.

(1) Yes, you’re right. inverse() is only available for GLSL >= 1.50. Since the Orange Book I think covers 1.40, it must have been a typo. Regardless, I have #version 150 in the source so it should be fine? The compiler doesn’t complain about not supporting 1.50, but it does emit that warning which I don’t understand. How does one select a profile?

(2) You are right, the determinant of a 2x2 matrix is the difference of two products, not the sum. Nevertheless, changing that didn’t seem to have any effect (though it should have been subtraction anyways).

(3) You’re right that it isn’t efficient to calculate in the shader. I couldn’t find a simple way of doing it via CPU and looked for some OpenGL functions, but most of the matrix routines are deprecated now for GL3. If I can find a simple way, I will provide it via a uniform variable.

(4) I don’t think it makes a difference, if I remember my linear algebra, whether you take inverse transpose or the transpose of the inverse. They should yield the same result.

I have #version 150 in the source so it should be fine? The compiler doesn’t complain about not supporting 1.50, but it does emit that warning which I don’t understand. How does one select a profile?

There’s no such thing in GLSL as a “profile”. That’s a concept that NVIDIA has with their Cg compiler, which they use as a GLSL compiler. That means that some GLSL error messages on NVIDIA platforms will refer to profiles. Just ignore it.

This is an NVIDIA driver bug. If they don’t support version 1.50, then the compiler should fail when it hits the #version directive. Since it does not, and inverse is part of 1.50, it is a driver bug for it to refuse to compile this. File a bug report on it.

Hey Alfonse. Thanks for your reply. It seems as though it is a bug and I will file it now. ugh