Mouse raycasting

Hi! I really struggle with implementation of mouse raycasting, so could you please help me?

Final point is counted incorrect if I rotate a camera around it origin (for simplicity it’s {0, 0, 0})

Implementation of camera matrix:

class Camera {
	ViewMatrix() {
        final_matrix_ = glm::translate(glm::mat4(1.0f), position_ + origin_);
        final_matrix_ = glm::rotate(final_matrix_, glm::radians(rotation_.x), {1, 0, 0});
        final_matrix_ = glm::rotate(final_matrix_, glm::radians(rotation_.y), {0, 1, 0});
        final_matrix_ = glm::rotate(final_matrix_, glm::radians(rotation_.z), {0, 0, 1});
        final_matrix_ = glm::scale(final_matrix_, scale_);
        return glm::translate(final_matrix_, -origin_);
	}
	ProjectionMatrix() {
		return glm::perspective(glm::radians(fov_), (float)window.x / window.y, 0.1f, 100.0f);
	}
};

Implementation of computing direction of ray:

glm::vec3 to_real_world_coords(glm::vec2 point) {
    glm::vec4 ndc_coords = glm::vec4(to_ndc({point.x, point.y, -1.0f}), 1.0f);
    glm::vec4 eye_space = glm::inverse(camera->ProjectionMatrix()) * ndc_coords;
	return glm::inverse(camera->ViewMatrix()) * glm::vec4(eye_space.x, eye_space.y, -1, 0);
}

Implementation of intersection (I know for certain, that ray hits sphere)

auto rotation_ = camera_->GetRotation();
auto rotation_matix = glm::rotate(glm::mat4(1.0f), glm::radians(rotation_.x), {1, 0, 0});
rotation_matix = glm::rotate(rotation_matix, glm::radians(rotation_.y), {0, 1, 0});
rotation_matix = glm::rotate(rotation_matix, glm::radians(rotation_.z), {0, 0, 1});
// camera position with rotation around 0, 0, 0
glm::vec3 origin = rotation_matix * glm::vec4(camera_->GetPosition(), 1.0f);

glm::vec3 direction = glm::normalize(math::to_real_world_coords(cursor_position));
glm::vec3 vertex_center = model->GetTransformation() * glm::vec4(vertex.position, 1.0f); // cast vertex from local to global
auto new_position = origin + direction * glm::dot(vertex_center - origin, direction);

auto new_local_position = glm::inverse(model->GetTransformation()) * glm::vec4(new_position, 1.0f); // cast result from global to local
model->SetVertexPosition(info.VertexId, new_pos);

Update, I tried differrent methods of calculating ray direction, and all of them gived the same result, so I think problem is with ray origin, but i still can’t figure out what’s wrong

Use the viewport transformation to convert from window coordinates to NDC (bearing in mind that the mouse Y is probably referenced from the top edge, increasing downwards).

float win_x = mouse_x + 0.5;
float win_y = win_height - mouse_y - 0.5;
// NDC x,y
float x = (win_x - viewport_x) / viewport_width  * 2 - 1;
float y = (win_y - viewport_y) / viewport_height * 2 - 1;

Then transform the points [x,y,-1,1] and [x,y,1,1] by the inverse MVP matrix, e.g.

glm::mat4 inv_mvp = glm::inverse(projection * modelview);
glm::vec4 A = inv_mvp * glm::vec4(x,y,-1,1);
glm::vec4 B = inv_mvp * glm::vec4(x,y,1,1);

Then convert the transformed points from homogeneous coordinates to Euclidean coordinates (i.e. divide by w). Those are two points on the ray in object space; specifically, the intersection of the ray with the near and far planes. Subtract one from the other to get the direction.

glm::vec3 P = glm::vec3(A) / A.w;
glm::vec3 Q = glm::vec3(B) / B.w;
glm::vec3 origin = P;
glm::vec3 direction = Q-P;

Hi! Thanks for the answer! I somehow managed to fix my own code, by changing order of rotation matrix in camera position:
From

auto rotation_matix = glm::rotate(glm::mat4(1.0f), glm::radians(rotation_.x), {1, 0, 0});
rotation_matix = glm::rotate(rotation_matix, glm::radians(rotation_.y), {0, 1, 0});
rotation_matix = glm::rotate(rotation_matix, glm::radians(rotation_.z), {0, 0, 1});

to

auto rotation_matix = glm::rotate(glm::mat4(1.0f), glm::radians(rotation_.z), {0, 0, 1});
rotation_matix = glm::rotate(rotation_matix, glm::radians(rotation_.y), {0, 1, 0});
rotation_matix = glm::rotate(rotation_matix, glm::radians(rotation_.x), {1, 0, 0});

But unfortunately I don’t quite understang why this was a problem. So I will try your solutions as well