I know it’s very dumb but I cant get around this.
I need to calculate the position of the camera (eye) in object space, given a transformation matrix M.
I think in theory is should be:
eye_obj = col * (V.transpose).inverse * M.inverse
Where:
col = {0,0,0,1}
V = 4x4 camera transform matrix retrieved via glGetFloatv(GL_MODELVIEW_MATRIX, V)
M = 4x4 transform matrix defining the center of the object
So now just plug-n-chug. The eye-space vertex position (v_eye) you’ve got is (0,0,0,1) – the eyepoint. So plugging it in, this will effectively just extract out the translation component in INV_MODELVIEW.
You can multiply V*M and invert the result to get INV_MODELVIEW, or you can invert M and V separately, and multiply them in the opposite order to get INV_MODELVIEW. Then multiply that by the vector (v_eye) of (0,0,0,1).
Note that my text above uses OpenGL transform ordering (operator-on-the-left – i.e. Ax=y). It looks like you’re using the opposite in your notation (what you’d expect for C++), so just flip everything left-to-right. That said, looks like you already have the right answer, except for that transpose.