Skip LookAt() call and see what happens.
I can’t skip it, I will not see my model anymore and I can’t test anything.
I will check the math tomorrow.
I was in a hurry when I wrote previous comment, but now I’ll try to elaborate…
Well, you are looking an object at the coordinate-system’s origin, and you need LookAt() for positioning the viewer?
With all responsibility I claim that you need only one translation, and maybe a rotation, but not such sophisticated function like gluLookAt().
Let’s analyze your first code-fragment. You have first translated and then scaled your verteces. This transformation is stored in the matrix combined. But nothing is happening until glMultMatrixd() call. Than you call gluLookAt(). That function multiplies matrix at the top of the model-view matrix stack with certain (let’s call “LookAt”) matrix (right-side multiplication). Than you call glMultMatrixd() and the previous product multiply with matrix combined (again right-side multiplication). So, your viewing transformation is in fact the following matrix:
V = MV * LookAt * combined
I hope that MV (previous matrix at the top of model-view stack) is I, and can be excluded from the previous equation. Nevertheless, we still have LookAt which translates and rotates whole scene so that the viewer is in the coordinate-system’s origin and looking towards negative Z-axis. When you call gluUnProject(wX,wY,wZ,modelMatrix,…), modelMatrix is equal to V, and unprojection gives good results, but in wrong coordinate-system. Would inverting just combined matix return coordinates in object (local) space. Probably not.
Do you see how difficult it is? We need just a reliable formula to get appropriate near and fare planes for objects with size from 1e-6 to 1e+6.
Changing approach will need a complete code refactoring (GL_NORMALIZE to adjust normals) and it is not even straightforward to unproject screen points.
gluLookAt is not different from a translation + rotation at the end a 4x4 matrix is generated like using the tra+rot combination.
The only benefit I see is that you can keep (far - near) = modelSize.
It is not very difficult, and if it is important to you to enable loading very small or very big objects, you shouldn’t give up such fast.
My initial suggestion was to scale a model by multiplying coordinates while loading. Not to perform glScale() transformation. On that way you can use your previous method, and by dividing unprojected coordinates with scaling-factor you have the original coordinates.
I saw that you have started a new thread on the same topic, but I’ll answer here. Take a look at the following link
Maybe this will help you.
I gave a quick look to the article. It looks like it explains how to get better separation from the depths between near and far planes. I would like to find instead a formula to not stretch too much the distance between the two.