I am now debugging some legacy code that sets up modelview and projection matrices for an OpenGL scene.

The approach it uses is different from what is recommended in OpenGL Programming Guide. When perspective projection is used, the projection matrix includes also translation and scaling components, apart from the canonical perspective projection components. This is the result of the way 3D view parameters are defined in the application (if you have ever heard about PHIGS and its “view orientation + view mapping” approach, the code I am debugging does similar things).

My question is: are there any potential side effects of combining canonical perspective projection matrix (like that generated by glFrustum) with some auxiliary translations, rotations and scaling and using the result as GL_PROJECTION matrix, instead of putting all “non-projection” transformations into GL_MODELVIEW matrix?

I am experiencing some strange and complicated incorrectness of the perspective, so I wonder if it may be the result of doing as described above.

In my opinion, I think it should causes some problems. Just looking at normals transformation, these ones have to be transformed by the modelview matrix inverse transpose but in your projection matrix you set the modeview matrix…

What kind of incorrectnesses are your experiencing?

Well, as I’m still working with simple shapes like boxes, cylinders and line segments, I’ve noticed no visible problems with transformation of normals (even if I’ve also noticed this point in description of OpenGL pipeline).

The most annoying problem is that the perspective distortion looks as if the projection center (the eye) was located not at the center of the viewport but somewhere at the lower left (or lower right in some cases) corner or even beyound the viewport boundary. There is obviously some incorrectness in projection matrix calculations.

Maybe I should better try to provide a sample screenshot here…
Could you please try to get the image by this link: http://files.mail.ru/download/01661694474f073e4670af67e91b5eda - this is an ordinary box oriented more or less vertically however its perpsective distortion is not as expected for a symmetric perspective…

The problems you get by putting transformations in the projection matrix that “should” go in the modelview matrix is related to those things that depend on the modelview matrix but not the projection matrix.

When tranforming vertices from object space to screen space to display them, it doesn’t matter where you put your transformations. It is perfectly OK to put them in the projection matrix. However, fixed function lighting and fogging, for example, depends on the modelview matrix only. That means, if you put your transformations in the wrong place, lighting and fogging will not behave as desired.

A common problem that beginners have is that the fog in their scene is static and moves with the world. That happens because they put viewpoint transformation in the projection matrix, and so the fogging system cannot be affected by viewpoint transformations, which results in a static fog, and not a dynamic fog following the viewpoint movements.

But all this are just recommendations for the normal case. If your setup calls for putting rotations in the projection matrix to get the correct projection, then of course you should do that. It’s all about what you want to achieve, knowing what you do and knowing what side effect it has on everything else.

Thank you a lot, you have answered my inital question!

I am starting to realize that the problem with incorrect perspective does not result from violating some OpenGL rules about GL_PROJECTION matrix but rather results from incorrect rotation/translation components of this matrix…

…and I still have to find the origin of my problem with perspective.

As far as I undertsand, the default orientation of the perspective frustum (generated by glFrustum() call) is as follows: the eye is centered with respect to (left, top) and (right, bottom) points, so the frustum is symmetric. To make it non-symmetric, it is necessary to apply translation and/or rotation to the projection matrix - is it true?

No, the glFrustum call assumes the eye is located at (0,0,0). So it’s symmetric if left and bottom are the negated values of right and top respectively, otherwise it’s non-symmetric.

PS. If you want to introduce a pixel-skew term you’ll have to generate the 4x4 projection matrix yourself.

Thus if I still want it to be symmetric, I need to apply a translation first (before applying the perspective projection transformation) that moves the eye to ( (left + right)/2, (top + bottom)/2, 0) - is it true?

Have no idea about “pixel-skew term”, is it anything similar to what I need (an auxiliary translation to achieve symmetric projection frustum) - could you please explain it a bit more?

I’m not sure what your goal is. If you have access to the rotation matrix and translation vector then you can create your modelview matrix and set a symmetric view frustum by setting ( -left = right ) and ( -bottom = top).

If you only know the combined projection+modelview matrix you can try to split it up into separate projection and modelview matrices using RQ decomposition.

My goal is to get symmetric perspective frustum centered wtih respect to the eye.

The situation is as follows. In my code, after modelview transformation, I have the coordinate system oriented properly. However, the required viewing volume is still not centered around the CS origin. I prefer not to chgange this (I mean calculation of the modelview matrix), as it would affect big parts of the old code.

Thus, summarizing the previous posts, in order to get symmetric perspective centered with respect to the eye I need to translate the viewing volume so that after this translation I have ( -left = right ) and ( -bottom = top) in projection matrix calculation.
Going to try this now…

P.S. And what about “pixel-skew term”? I guess it’s the 5th component of the projection matrix (among 16), is it used to achieve any special result?

Applying a translation that centers the model between right and -right along X and top and -top along Y helps.

If canonical perspective projection matrix (calculated by glFrustum()) is

p1 0 p9 0
0 p6 p10 0
0 0 p11 p15
0 0 -1 0

then the modified matrix including the translation should be

p1 0 p9 p13
0 p6 p10 p14
0 0 p11 p15
0 0 -1 0

where
p13 = -p1 * cx/2,
p14 = -p6 * cy/2
cx = (xmin + xmax)/2 and cy = (ymin + ymax)/2, and xmin,xmax, ymin,ymax - boundaries of the model along X and Y axes.

Of course, I have to calculate the projection matrix manually.