I want to transform some vectors from view space back to object space. Usually I would use the ModelViewInverse to do this with points but how can I do this with normals? The normal needs to be transformed by the inverse transposed of a matrix. But what exactly is the inverse transpose matrix of the modelviewinverse matrix?
Right, normals are transformed by the inverse transpose of the modelview matrix:
n’ = (M^-1)T * n;
To reverse that transformation you actually need the inverse of that which is
n = ((M^-1)T)^-1 * n’;
not what you said the inverse transpose matrix of the modelviewinverse which is ((M^-1)^-1)T.
That’d be only equivalent if you have just rotations in the normal matrix! Which you normally should!
Translations do not affect vectors, so only the 3x3 matrix is needed.
The inverse of a rotation is its transpose.
n = ((M^-1)T)^-1 * n’; <=> n = ((MT)T)T * n’; <=> n = MT * n’;
Well, looks almost too simple to be true.
But makes sense because the inverse transpose of a rotation is the original rotation.
So it doesn’t matter if there are only rotations or not, multiplying the transpose of the upper-left 3x3 part of the modelview matrix with the normal should always work.