However, when the camera projection matrix is a projection matrix from OpenVR the function produces bad results. This is just the inverse of the same projection camera matrix being used to calculate vertex positions. I don’t see why it wouldn’t work when a conventional projection matrix does.
Any ideas why this is happening and how to fix it?
The (VIEWING*PROJECTION) takes you to CLIP-SPACE. So the input to the inverse transform is CLIP-SPACE (not NDC-SPACE). That may have something to do with it.
For something else to try, check out PositionFromDepth_DarkPhoton() at the bottom of this post. Though there’s all kinds of ways to fry this fish:
Yeah, the Leadwerks way works well for Leadwerks perspective projection matrices, and avoided the use of an extra buffer for screen positions. I switched to the matrix multiplication specifically to support VR, because I don’t know if all the different headsets will use a conventional perspective matrix.