Given a plane P = { a, b, c, d } and knowing the Projection and Modelview matrices, how do I determine if the plane’s surface is visible? (Because I don’t want to recurse into non-visible mirror faces.)

Is there a quick way of determining if the stencil buffer was written to while I’m stenciling out the mirror face area? (Because I don’t want to recurse into mirror faces that are all covered up.)

Are there performance issues in using either the PROJECTION or MODELVIEW matrices for performing arbitary matrix multiplication? (e.g. For A x B, PushMatrix, LoadMatrix(A), MultMatrix(B), GetFloatv(…, Result), PopMatrix);

I’ve worked out the answer to my first question now:

Where:
M = Modelview Matrix

[ 0]

V = View vector = M x [ 0]
[ 0]
[-1]
(Assuming the last row of M is [0,0,0,1])

Then:

The plane’s surface is visible if:

(M x P) dot V

is positive

I thought I might have to multiply the modelview matrix onto the projection matrix but that wrecks it because the last row becomes no longer [0,0,0,1].

As for my second question, the only way I can think of is by using ReadPixels() to read the stencil value accross the entire viewport… But this doesn’t seem very efficient.

The M x P part of the solution I posted above is wrong. It shouldn’t be M x P, it should be the normal of M x P. Just using M x P only works if the world isn’t sheared.

Unfortunately, it’s beyond my maths ability to think up a general equation for the normal of M x P. But, the P I’m using will always be (0, -1, 0, 0) so I can just get the cross product between the X column of M and the Z column of M.

yoyo, the NV_occlusion_query test works great --good speed improvement and I don’t have to worry about surface visibility maths. I used the wglGetProcAddress() function so I don’t know how portable my GLUT program will be now. …I don’t know what “ARB” means.

cp gives the camera’s position, p is the mirror’s plane.

Of corse there can be cases when you are in front of the mirror, but yet the mirror’s surface is not visible (outside of view frustum). This can be solved by testing the mirror’s surface against the view frustum. There is a good tutorial on how to do it at http://www.markmorley.com/opengl/frustumculling.html .

This way you can test the visibility using basic opengl functions, so your code remains portable. I don’t know how this influences speed but I think it’s not so slow…