Ray casting with a fixed camera toward a rotated model

I want to cast a ray into my model based on a mouse click and be able to select the closest intersecting point.

The difference I have, is that when “rotating”, I rotate the model, not the camera. My application is a model viewer, not a game, so I want the user to see the other sides of the model, not look around the world.

The code approach I’ve been using is from this tutorial http://antongerdelan.net/opengl/raycasting.html and is based on the following:

private Vector3f calculateMouseRay() 
    float mouseX = Mouse.getX();
    float mouseY = Mouse.getY();
    Vector2f normalizedCoords = getNormalisedDeviceCoordinates(mouseX, mouseY);
    Vector4f clipCoords = new Vector4f(normalizedCoords.x, normalizedCoords.y, -1.0f, 1.0f);
    Vector4f eyeCoords = toEyeCoords(clipCoords);
    Vector3f worldRay = toWorldCoords(eyeCoords);
    return worldRay;

Full ray cast code is here: https://www.dropbox.com/s/qkslys3p3xzh8av/MousePicker%20Code.txt?dl=0

My model rotation code is:

    QMatrix4x4 modelMatrix;
    modelMatrix.translate(xCenter, yCenter, zCenter);
    modelMatrix.rotate(m_zRot, QVector3D(0, 0, 1));
    modelMatrix.rotate(m_yRot, QVector3D(0, 1, 0));
    modelMatrix.rotate(m_xRot, QVector3D(1, 0, 0));
    modelMatrix.translate(-xCenter, -yCenter, -zCenter);

My camera is set at:

    QVector3D cameraPos(m_xMax*1.25, m_yMax*0.75, m_zMax);
    QVector3D cameraTarget(xCenter, yCenter, zCenter);
    QVector3D cameraUp(0,1,0);
    m_camera->lookAt(cameraPos, cameraTarget, cameraUp);

What I want to do is draw a line from the mouse click through the scene.

I know I need to de-embed the model rotation, by multiplying by the inverse matrix somewhere along the chain, but thus far I haven’t been successful.

Where amongst the first steps listed above should I de-embed the model rotation to get the ray cast vector?

Maybe I’m missing something here, but to construct a line you need two points, not one.
The matrix rotating the model is only meaningful when projecting the model’s points into world space.
So, first you can construct a line from the mouse-pointer through the scene by un-projecting two points (mouseX,mouseY,someZ) and (mouseX,mouseY,anotherZ) and get the two points of the line in world-space.
If you then want to check for intersection with the model you either

  • project the two line-points from the first step into model-space, by multiplying them with the inverse model-matrix, and test for intersection with the mesh in model-space
  • project the mesh (or some of it’s triangles) into world-space and check for intersection there.

A point in object space is converted to a point in window space by a sequence of transformations, for example:

model matrix -> view matrix -> projection matrix -> viewport transformation

The viewport transformation is derived from the parameters to glViewport() and glDepthRange() rather than allowing an arbitrary transformation matrix, but it can still be expressed as a matrix.

What you need to do is to compose all of those transformations into a single matrix, invert it, then transform the points [x,y,0,1] and [x,y,1,1] by the result. Both of the resulting points lie on the ray.

Also, bear in mind that mouse coordinates are usually reported relative to the top-left corner of the window, while OpenGL’s window coordinates are relative to the bottom-left corner.

Thank you both.

I finally solved it by essentially copying the code from glm::unProject().

I call it once with mouseZ = 0, and a second time with mouseZ = 1, that yields the x/y mouseclick on the near plane and far planes respectively. I connect the two points with a line, and the line draws through my scene exactly as I’d expect, for all rotations.

Next up is to do some vertex checking along the way to find the intersection point…

Thanks again. :slight_smile:

model matrix -> view matrix -> projection matrix -> viewport transformation

Don’t forget the perspective division step. It’s between projection and viewport. Also, since its a non-linear transform, it can’t be done via a matrix.