fog depending on modelview matrix

Hi all,

I kindof discovered that OpenGLs fog depends on the actual set modelview matrix.

My case:
I render some object with fog enabled. Actual modelview matrix is identity - object & fog looks fine. If I replace the actual modelview matrix with some perspective projection matrix (shadow projection) the objects renders just right (now looks like the shadow of the object:) but it is wrong ‘fogged’.

(Hope thats not too ill formulated :D)

One could think of doing the fog in a vshader or premultiply the object with the matrix on the cpu but that are no options :stuck_out_tongue: Had some google searches with nice fog tutorials coming up. Redbooks fog chapter said that fog calc is done after transformation/texturing/… but nothing that helped me understand the dependance.

Would be nice if anyone could enlighten me. Big thanks for your help!


The simple fog model that OpenGL employs is based on a distance that’s calculated as either the eye-space vertex z or the actual Euclidean distance from the eye to the vertex. I believe the default is the eye-space z, which is fast but depends on the modelview matrix and is therefore view-dependent. If you set the fog hint to “nicest” you might get better results; you can look for a “radial” fog extension which will always use the actual distance and will therefore be view-independent, which is usually a good thing.

The projection matrix does not fit into either of these models, however. Look at the fog equations in the OpenGL specification and it will become clear as to why this is so. The distance calculation is key here: eye-z or distance.


Your mistake is placing a perspective transform into the modelview matrix. It belongs into the projection matrix.

The gl fog is calculated from the distance of a vertex to the eye (the origin) after transformation by the modelview matrix. This requires the modelview matrix to produce view coordinates. When you put the projection part onto the modelview matrix, you’re screwing with that assumption, as you actually produce normalized device coordinates.

Also, it’s not only fog that’s affected adversely. Lighting and user clip planes will break, too.

Thanks for your help. The problem is clearer now. I tried to multiply the shadow projection matrix with the actual set projection matrix. Well, its works but now the shadow itself appears to be view-dependant.

the current operation order looks something like this:

// setup projection and viewing


// render object ‘normal’ way

// ‘render’ object with shadowProjMatrix to create shadow




Do I miss something or is it just my math? (Setting the shadowProjectionMatrix in the modelview looks fine - with the fog/userclipplanes problems of course. Setting it in gl’s projection matrix the fog & stuff looks fine but the shadow itself is wrong.)

If you’re using projected shadow geometry, then you want that transformation on the modelview, not the projection stack. Think of it as a model transformation the flattens the model onto the shadow plane. The current view transforms should be left alone for this.

So you’d just need to change this bit:

// 'render' object with shadowProjMatrix to create shadow
// Modelview is current here

Fogging the shadow itself is another matter, and will work the way any other fogged object would.

Uh, now i’m a bit confused :smiley:

The code snippet that you posted shows the way I had before. This was the case when fogging the shadow did not work properly - the reason why I tried to move the shadowProjection matrix in gl’s projection matrix.

Of course, when you move the position where you apply the matrix, it changes the output. You should, however, be able to change the shadow projection matrix such that you the end result will be the same as before.

What you had before was

v' = P * (V * S * M) * v
     ^   ^^^^^^^^^^^ 
     |    modelview

What you need is

v' = (P * S') * (V * M) * v
     ^^^^^^^^   ^^^^^^^
        |      modelview

so that you can put the shadow projection onto the projection matrix.

    P * S' * V * M = P * V * S * M
<=> S' * V = V * S
<=> S' = V * S * V^-1

So, instead of multiplying the normal shadow-projection matrix onto the modelview stack, you can push the product of view, shadow projection, and inverse view matrix onto the projection stack.

DISCLAIMER: this is totally untested

EDIT: As the contents of the model view matrix and vertices are unchanged to the normal object draw pass, the fogging result for each vertex will be the same as for the original geometry. This may or may not be what you intended.

Putting it down with the matrices like you did helped me a lot. And your suggested solution works great.

Thank you very much (all of you)!