I guess, this is more a math-problem. But since the math forum is hardly visited, I try my luck here…
I have to generate a projection matrix for picking. This matrix builds the frustum around a pixel at a given 2D viewport coordinate. Until now I could generate this matrix from knowns like fov, aspect ratio, near and farplane (symmetric projection), just like I would use glFrustum().
Now I´m facing the problem that I do not know aspect, fov etc. anymore. The only thing I got left is a projection matrix (arbitrary; could be projective, orthographic symmetric or asymmetric)that I get handed over. The challenge is to manipulate that given projection matrix in order to achieve the same effect (i.e. bound the frustum around the pixel that the user clicked on.)
I´m not good at maths in homogeneous coordinates. But I was hoping that I could easily “move” the frustum-planes in clipspace by just manipulating some of the values in the projection matrix. Any hints are welcome.
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPickMatrix(x, viewport[3] - y, pickWidth, pickHeight, viewport);
gluPerspective(...); // or whatever your projection matrix setup is.