However, when the camera projection matrix is a projection matrix from OpenVR the function produces bad results. This is just the inverse of the same projection camera matrix being used to calculate vertex positions. I don’t see why it wouldn’t work when a conventional projection matrix does.
Any ideas why this is happening and how to fix it?
Yeah, the Leadwerks way works well for Leadwerks perspective projection matrices, and avoided the use of an extra buffer for screen positions. I switched to the matrix multiplication specifically to support VR, because I don’t know if all the different headsets will use a conventional perspective matrix.