I need to render some objects in my scene in orthographic projection and be able to interact with them (I am using ray casting to get intersection points). Instead of using an ortho projection I had the idea that it should be possible to change the corresponding transformation matrices in such a way that they ‘undo the perspective projection’. But I don’t know if it is possible at all (although I think it should be) and how to calculate these matrices. Any help is welcome.

Can you explain why you want to “undo the perspective projection”?

Like all matrices, you set the projection matrix, if you don’t set perspective projection, projection won’t be perspective projection, but orthographic projection, if I am right.

What I mean is that, I don’t understand why you want to undo a thing that you have to do, just do nothing…

It would be easier to just calculate a different transformation matrix than to change to orthographic projection in the render framework I am using.
I know of course, that I can do all the stuff with a real orthographic projection but programmers are lazy. The objects I need to render orthographic are positioned in the scene like all other (perspective projected) objects and I thought just playing around with their transformation matrices would be an easy and clever way.

It would be easier to just calculate a different transformation matrix than to change to orthographic projection in the render framework I am using.

What do you mean?

I think I don’t understand what is your point but if you want to switch easily to an orthographic to a perspective projection and or in the opposite way I think you better set up two projection matrices, one for perspective projection the other for orthographic projection and then, set GL_PROJECTION matrix the one you need.

I think you mean two projection and not texture matrices.
And you are correct that using two matrices and switching as needed would solve my problem. It is just that the render framework does not allow for arbitrary switching of projection matrices as it is only intended for perspective projections. I can workaround this and do the ‘normal’ ortho/perspective switching as you suggest and it will solve my problem. I know how to do it, so no problem there.
The question was if it is possible to change the transformation matrix of an object in such a way, that rendering it with perspective projection will result in an orthographic rendering of the object.
Some years ago, when shadows were a really big problem you did planar shadows by constructing a transformation matrix that projected the objects vertices on a plane (flattening the object) and drawing that as your shadow. I was thinking along these lines when I came up with my idea, only that you would construct a matrix that holds something like the inverse of the actual projection matrix so that rendering your object with this matrix would result in an orthographic projection of only this object with no need to touch the GL_PROJECTION matrix at all.
Perhaps it is a stupid idea, so I will just take the standard route of doing this stuff.

I think you mean two projection and not texture matrices.

Yes, sorry I was thinking about a texture matrix before reading your post.

I was thinking along these lines when I came up with my idea, only that you would construct a matrix that holds something like the inverse of the actual projection matrix so that rendering your object with this matrix would result in an orthographic projection of only this object with no need to touch the GL_PROJECTION matrix at all.

Ok but where would you put this matrix if not in the GL_PROJECTION matrix which is used for projection. I don’t think you could do that using the fixed pipeline. Using a shader you can do whatever you want and use none of the opengl matrices giving some customs matrices to the shader. But maybe it is not possible in your render framework.

I wanted to put it into the transformation matrix of the object to have something like an inverse perspective object space. The projection matrix is just multiplied to the rest of the matrix bunch like any ordinary matrix (verticies get first multiplied by modelview then by projection after that there is the perspective divide etc…). The idea was to multiply my special matrix with the modelview matrix so that the following multiplication with the perspective projection matrix would be ‘canceled out’ so that you end up with an orthographic projection. An advantage would be that I could just transform my rays to the now deformed object space and do all intersection tests as normal.
But I must admit that I forgot about the perspective divide which must be canceled by the new matrix, too. Thinking about it, it sounds more complicated than I first thought.
As said I think I can workaround the ‘no projection matrix change during a frame’-limitation and use the normal approach. Shaders would be an option, too.
But nonetheless I am interested to know if my idea could have worked or if it is totally rubbish.

I don’t think there will be a problem with perspective division, this is not part of any matrix and with ortho projection, w will always be 1. But there could be problems when transforming normals for lighting calculations which are transformed by the the modelview inverse transpose matrix…

There was another post where they talk about putting all transformations from model to projection in the modelview matrix, I will put here a link when I find it.

Thanks for the link but it did not really help. I think the real question is if the perspective projection matrix is invertible. This I can figure out myself. Thanks for the help.

The perspective matrix should be invertible unless you put some exotic projection matrix in my opinion. Anyway you can check this computing its determinant.