The matrix m has initial value equal to glm::mat4(1.0f) which is the identity matrix, and then it’s updated by rotation in this way: m = glm::rotate(m, radians, axis); and I’m trying to get the basis vectors up, right and forward from it. How should I do that?
I thought this was a simple question at first but then ran into troubles. First, I thought I can take the 1st, 2nd, 3rd column directly as right, up and forward, but since OpenGL assumes a right-handed coordinate system, this is already wrong in the first step, because if z axis is forward and y axis is up, then the first column of identity matrix is “left” rather than “right”.
I know I can also compute the basis vectors by applying m on the initial basis vectors, or using euler angles with sin and cos, for robustness, I finally decided to use quaternions and it does give me the correct result. However, I realized that some 3rd-party libraries only takes in matrix m as parameter, and then assume that the first 3 columns correspond to right up and forward, I’m now using ImGuizmo to draw gizmos which does exactly that, as a result, the gizmo is not drawn correctly as I would expect.
Now of course there are ways to fix this, but I wonder if the way I use matrix m is correct. Ideally, I want the first 3 columns of m to be always right up and forward, but that would require me to initialize m to a non-identity matrix. What should I do? Do I need to convert handedness of coordinate system?
GLM defaults to using right handed coordinate system conventions, matching what OpenGL does. You can change the convention used by GLM by defining GLM_FORCE_LEFT_HANDED - note that the resulting matrices will no longer be directly usable with OpenGL.
If you use a left-handed coordinate system, the matrix will have a negative determinant. This effectively turns the geometry inside-out (clockwise faces become anti-clockwise and vice versa), so you may need to change the glFrontFace setting to account for this.
Also, bear in mind that OpenGL is ultimately just doing calculations on numbers. Any physical interpretation is up to the user. Typically, negative Zeye is considered forward because a) the conventional projection matrices (created by e.g. glFrustum) result in ZNDC having the opposite direction to Zeye), and b) the initial state for glDepthRange is (0,1) and that for glDepthFunc is GL_LESS, so fragments with lesser ZNDC values obscure those with greater values, i.e. greater ZNDC is “farther” from the viewpoint.
Thank you for your solution. I followed your advice and changed my forward to be negative Z now. I realized that NDC space is left-handed anyway and I’m free to choose the world space handedness. Reversing the front face is a very nice trick without needing to change my matrix, but in my case I can’t reverse face in vendor libraries, so I used glm::scale(1,1,-1) to get the job done.
Just another question though, how did you type the subscript after a letter? like Z_NDC and Z_eye? Can I use latex syntax here? I can’t figure out how to type it correctly.