How to get a ray vector at the mouse position?

hi all, probably this question has being asked in some other form. I am trying to do some ray tracing stuff. All I want is the index of the vertex on the 3D graphic that is hit by (or is nearest to) the ray pointing through the mouse position into the screen. I already have written the routine to calculate distance between a vector and a 3D point. My graphic consists to about 2 million vertices, but I can do calculations fast enough using CUDA. But I do not know how to get the vector that defines the ray. I need the origin of the ray (as a 3D vertex) and the direction vector. how can I get these parameters? I would appreciate any help in this regard.

Thank you in advance.


PS: Here is what I have found so far. I can use the following code to get the mouse position on the near Clipping plane in GL coordinates

int ViewPort[4];
double modelVw[16];
double projectionVw[16];
float winX, winY, winZ;
double posX, posY, posZ;
glGetDoublev(GL_MODELVIEW_MATRIX, modelVw);
glGetDoublev(GL_PROJECTION, projectionVw);
glGetIntegerv(GL_VIEWPORT, ViewPort);
winX = (float)X;
winY = (float)ViewPort[3] - (float)Y;
winZ = 0;
glReadPixels(X, (int)winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);
gluUnProject( winX, winY, winZ, modelVw, projectionVw, ViewPort, &posX, &posY, &posZ);

But I get messed up results when I print out the values of posX, posY and poSZ. Qnan and #Inf are the values I get quite often.

here is the minimal code I am using, during initialization:

gluPerspective(45, 1.0, 0, 1000);

gluLookAt(0,0,-100, 0,0,0, 0,1,0);
and then I call the above code to find the value of the mouse co-ordinates. But I get the value -1.#QNAN, -1.#QNAN, -1.#QNAN for posX, posY and poSZ. I have programmed to perform this function every time I double click the window. On second time double click I get the values: -3.18285e+048, 0.000601179, 2.05309e+077. these values do not change on any of the next double clicks. Anybody with any idea, what i could be going wrong?

Thanks in advance

Hi, Vj,
When you call gluUnproject, please make sure winZ is not equal to 0; winZ is the depth value in DEPTH_ATTACHMENT in framebuffer of the point your mouse clicked.

You can call
glReadPixels(winX, winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &d);
to get it.

Hello ayongwust_sjtu,

Thank you for your reply. Yeah I use readpixels() to get value of winZ. I also tried printing out the value for winZ which turns out to be 0.9… everytime. But I am still getting weird numbers for posX, posY and posZ.


Hi Vj,

It may be related to your gluPerspective() call : the near plane should be strictly > 0 (from the man page : Because r approaches infinity as zNear approaches 0, zNear must never be set to 0).

Note about gluUnproject : there is no problem having the 3rd parameter equal to 0, these are window space coordinates (depth value range is [0.0, 1.0]), not object space.

Hello Eric,

Actually I am using zNear = 1.0. Sorry for the confusion in my first post. 1.0 or any other value for zNear does not make any difference.


hey Guys,

I think I figured out what was wrong. Intellisense completed something for me, and stupidly I missed it. I was using glGetDoublev(GL_PROJECTION, projectionVw) instead of glGetDoublev(GL_PROJECTION_MATRIX, projectionVw). hopefully I should get the expected results now.


Also missed it !