Normally I’d suggest the OpenGL selection mechanism, for its simplicity. But that’s fallen out of favor since its deprecation.
Tracing a ray is easy enough. In a nutshell what you want to do is minimize 2 quantities:
- the orthogonal distance to the ray from the vertex (vertices closest to mouse in XY) and
- the distance from the ray origin to the vertex along the ray (vertices closest to mouse in Z)
Let V be the vertex position under consideration, O the ray origin and the D the normalized ray direction.
The orthogonal distance for (1) is then
dist_xy = length(V - (O + dot(V - O, D) * D)),
and the z distance for (2) is
dist_z = dot(V - O, D).
So for each vertex considered, keep the one with the smallest xy and z distances. There are lots of ways to optimize this and plenty of code out there, but that’s the basic idea. I would recommend any good book on computation geometry and ray-tracing for a deeper look. The oft referenced Real-Time Collision Detection does a great job distilling a lot of the more useful algorithms and offers some great tips on how to avoid common floating point pitfalls and such.
Hope it helps.