I need to rotate a polygon so that it lays only in one plane (ie XY). I need to do this, because only then i can calculate the texturecoordinates in my editor.
I know about matrix-multiplication and i am not that bad at vector maths, but i don´t think my approach will ever work. So i would like to ask if someone could explain it to me.

I think what I would do is the Gram-Schmidt orthogonalization process. Your polygon has some kind of vertices in a plane I would imagine. The plane should be spanned by two vectors. In the xy-plane, the spanning vectors would be {(1,0,0),(0,1,0)}. All the points of your polygon will be linear combinations of these two vectors. Say one point is (c1, k1,0) = c1*(1,0,0)+k1(0,1,0)

If you wish to transform your polygon to a different plane, perform the gram-schmidt process to get two orthogonal vectors for your new plane. Say your new spanning vectors are v1, v2. Your polygon in the new plane is the same linear combination of the new vectors. So your first point (c1,k1,0) goes to c1v1 + k1v2 in the new plane.

The entire process should be explained in detail in an (advanced?) linear algebra book.

I hope that helps.

Originally posted by Jan2000:
[b]Hi everyone.

I need to rotate a polygon so that it lays only in one plane (ie XY). I need to do this, because only then i can calculate the texturecoordinates in my editor.
I know about matrix-multiplication and i am not that bad at vector maths, but i don´t think my approach will ever work. So i would like to ask if someone could explain it to me.

Well you should be able to work out the axis of rotation and the angle pretty easily between the 2 planes. This is a quaternian (in effect) and you can make a transformation matrix from this and just use glMatrixMult.