Mouse projection failed


I’m trying to project my cursor in 3D world since few days and I’ve got some problems to detect ray/box intersection.
To present my problem, I will show you how I proceed:

First, I create my ray from my camera informations:

const glm::vec3 position    = glm::vec3( mouse.x, screenH - mouse.y, 0.f );
const glm::mat4& view       = camera.getViewMatrix();
const glm::mat4& projection = camera.getProjectionMatrix();
const glm::vec4& viewPort   = camera.getViewport();

ray.origin    =  glm::unProject( glm::vec3( position.x, position.y, 0.f ),
                                            viewPort );

ray.direction =  glm::unProject( glm::vec3( position.x, position.y, 1.f ),
                                            viewPort );

After that, I compute bounding box for every model and I test intersection :

BoundingBox box;

if( box.intersectLine( ray.getOrigin(), ray.getDirection() ) )
      // …

The problem appear when mouse is “far” from center, I think problem is here because I compare my ray (in world space) to my models (in local space),
[li]How can I transform the result of glm::unproject to local space? [/li][li]Am I using “glm::unproject” correctly? [/li][/ul]
I’ve tried to draw the computed ray, it is correctly draw where I click.

Thanks a lot for your time!

Transform both points by the inverse of the object’s transformation. I’m assuming that each object has a separate transformation, otherwise this should be accounted for by camera.getViewMatrix().

If you’re desperate to do it without using any additional matrix functions and don’t care about performance, you can just execute your first block of code for each object, with the object’s transformation incorporated into the camera’s view matrix.

But going forward, you’re better off performing all of the matrix operations in the application, and just passing composed results to GL with glLoadMatrix() or glUniformMatrix(). With OpenGL 1.x, it’s more efficient, with OpenGL 3.x core profile, you don’t have a choice.

Yes, assuming that glm::unProject() and the camera methods are trivial wrappers around the equivalent OpenGL functions (i.e. same argument order, same matrix storage convention, etc).