Terrain 3D position from mouse coordinates


I have big problem, I don’t know why gluUnProject returning NaN values, so I trying to build own gluUnProject, but is failing.

Target platform: Crossplatform
Framework: Qt5
Library: QOpenGL, not QGL!
OpenGL ver: 4.0+
Current OS: Windows 8

So I’m trying to get terrain coords from my own camera class with QMatrix4×4, but it returns incorrets 3D values.
Im using in camera these properties:
viewMatrix, projectionMatrix and viewProjectionMatrix.
I tried to figure it out with this code:

[li][FONT=arial] QMatrix4x4 inverted [COLOR=#363534]= m_camera->projectionMatrix() * (m_camera->viewMatrix() * m_modelMatrix);[/li][li] inverted = inverted.inverted();[/li][li] float posZ;[/li][li] [COLOR=green]// m_funcs = QOpenGLFuncions_4_0_Core[/COLOR][/li][li] m_funcs->glReadPixels(mouse_position.x(), mouse_position.y(), 1, 1,GL_DEPTH_COMPONENT, GL_FLOAT, &posZ);[/li][li] QVector4D clickedPointOnScreen(mouse_position.x(),mouse_position.y(), posZ, 1.0f);[/li][li] QVector4D clickedPointIn3DOrgn = inverted * clickedPointOnScreen;[/li][li] terrain_pos = clickedPointIn3DOrgn.toVector3D();[/li][li] [b]qDebug/b << terrain_pos;[/li][/ol]
I’m with camera on position: 250.0f, 10.0f, 250.0f
And I’m getting in result: QVector3D(414.76, 105.6, 6.37669) (cursor is in center of QGLWidget screen [935, 475], i calculated coords). It should be QVector3D(250.0f, 10.0f, 260f±)

To calculated coords: I have fullHD = 1920x1080, but QGLWidget is only 1870x950
To QGLWidget: I’m using it because QtGui cannot have controls which I needed (making editor)

Edit: Camera is perspective mode!
Edit2: posZ is often 0.9xxx, maximum 1

It’s probably dividing zero by zero, which probably indicates that the arrays being passed are incorrect.

You need to concatenate the viewport, projection and model-view transformations (in that order: VPM), invert the result, transform the window coordinates by the resulting matrix, then divide by w to convert from homogeneous to Euclidean coordinates.

Note that the Z coordinate needs to be converted from the 0…1 range returned by glReadPixels() to the -1…+1 range, either explicitly or as part of the viewport transformation (gluUnProject treats this as part of the viewport transformation). Also note that OpenGL window coordinates have (0,0) in the lower left while GUI toolkits typically report mouse coordinates with (0,0) at the upper left.

If gluUnProject() isn’t working because you’re feeding it incorrect data, feeding the same incorrect data to your own version won’t work any better.

Z coordinate now returning 5.67406e-39 That’s bad right? Or not? (Changing every time u run app ex to: 1.28372e-38)
Yes viewport lower left is 0, 0

Currently coords at (935, 475) QVector3D(250, 9.99991, 249.8)
But if you will see screen it’s still not correct. Link to image. On screen its alot away! It should be QVector3D(250, 10, 230)

Final code: (Forum remove whitespaces :confused: pastebin: http://pastebin.com/rReETsJC )


modelViewProject; // m_viewportMatrix is from resizeGL from calculated glViewport







EDIT: when I remove in glReadPixels posY to mouse_position.y() it gives right coords when I arrive to position on nearClip, but from farClip it’s still wrong :confused: I think, here is problem with posZ


I have solved why posZ returning wrong value, it was because posY was width() - mouse height!

But I have still trouble with final Vector3D result! Still Vector3D.Z returning wrong value -20f

Current source code: http://pastebin.com/TX4RYmNM

Currently QVector3D on [935, 475] = QVector3D(249.997, 10.0002, 216.095)
Should be QVector3D(250, 10, 230), I don’t care about X, Y ± 0.0003

New discovering…

When I rotate camera in axis X to left by 90 degrees, my cursor Z axis is now correct, but cursor X have incorrect values, same with axis Y, when I rotate camera on axis Y to -90 degrees I get -values.

Any help to this? What’s wrong? Some calculation with Matrix? How it can be fixed? Same code as preview post.

This is my last problem, please let comment if you have any idea!