I use gluUnProject to get the ray perpendicular to a pixel of the screen :
gluUnProject(x, viewport - y, 0, modelviewMatrix, projectionMatrix, viewport, &x1, &y1, &z1)
gluUnProject(x, viewport - y, 1, modelviewMatrix, projectionMatrix, viewport, &x2, &y2, &z2)
In the render function, I use an ortho projection :
glOrtho(-2, 2, -2, 2, -2, 2)
I noticed a strange return of the gluUnProject function when I put a z-translation of the objects in the render function :
glTranslated(0, 0, -1)
=> I get : z1 = 3, z2 = -1
I would have expected the following result : z1 = 2, z2 = -2 because it is the bounding box of the ortho projection.
I get this expected result for x1/x2 and y1/y2 when I put a x- or y-translation.
Any idea why the z-axis seems different ?
Calling glOrtho(-2, 2, -2, 2, -2, 2) will squeeze everything in that “bounding box” onto your viewport. However, that is only for things that fall into the box. If I have a vertex coming in at glVertex(0,-10,0) and the identity on the modelview matrix, it will not be seen. However, if I call glTranslate(0,10,0) on the modelview matrix, it will push the vertex into the said box and it will be on screen. gluUnProject will then take the two matrices you provide it and push it back to it’s original space, which is at (0,-10,0), which is obviously outside the ortho box you mentioned.
It’s all about spaces. Right now gluUnProject is going from window space back to model space, how the vertices were first sent in. Say, for example, you wanted “camera” space the modelview would be the identity. Or if you wanted “world” space, the modelview matrix you pass to gluUnProject would be just the camera matrix (like the one gluLookAt provides) without the glTranslate for the specific object.