Rotations about arbitrary axis

I`m struggling to come to grips with rotations about arbitrary axes.
I want to create the “flying camera” effect. There are 3 axes, x, y and z while “x” is horizontal, “y” vertical and “z” points forward. I want them to be arbitrary, so when camera is being rotated around “x” axis it rotates around horizontal axis (horizontal - in this case, it means parallel to application window). To achieve this goal I did some maths…

glPushMatrix();
glRotatef(zCamAngle, 0.0, 0.0, 1.0);
glRotatef(yCamAngle, s(zCamAngle), c(zCamAngle), 1.0);
glRotatef(xCamAngle, c(yCamAngle)+s(zCamAngle), c(zCamAngle), -s(yCamAngle);
glPopMatrix();

Where, s and c are sin and cos respectively. However, there is an error in last glRotatef line. I have no idea what arguments should be placed there.
Is there any easier way to deal with this problem?

Something like gluLookAt ???

Sounds like you want to transform the view relative to where it is pointing right now.

The trick I use to do this sort of stuff is to peek into the modelview matrix to get the direction of each axis.

The top left 3x3 part of the modelview matrix is actually three vectors that point in right/up/forward. If you rotate around that Z axis, you’d tip left/right, if you rotate around that Y axis, you’d look left right, if you rotate around that X axis, you’d look up/down.

Here is code that will give you a ROW MAJOR rotation matrix about an arbitrary axis (it needs to be transposed before handing it to OpenGL):

	MatrixT &LoadRotateAxis(const VectorT<T> &axis, T angleRads)
	{
		T c, s, t;
		VectorT<T> sv, tv;

		s = (T)sin(angleRads);
		c = (T)cos(angleRads);

		t = 1.0f - c;

		sv = axis * s;
		tv = axis * t;

		m[_M44(0,0)] = tv.x * axis.x + c;
		m[_M44(0,1)] = tv.x * axis.y - sv.z;
		m[_M44(0,2)] = tv.x * axis.z + sv.y;
		m[_M44(0,3)] = 0.0f;

		m[_M44(1,0)] = tv.x * axis.y + sv.z;
		m[_M44(1,1)] = tv.y * axis.y + c;
		m[_M44(1,2)] = tv.y * axis.z - sv.x;
		m[_M44(1,3)] = 0.0f;

		m[_M44(2,0)] = tv.x * axis.z - sv.y;
		m[_M44(2,1)] = tv.y * axis.z + sv.x;
		m[_M44(2,2)] = tv.z * axis.z + c;
		m[_M44(2,3)] = 0.0f;

		m[_M44(3,0)] = 0.0f;
		m[_M44(3,1)] = 0.0f;
		m[_M44(3,2)] = 0.0f;
		m[_M44(3,3)] = 1.0f;

		return *this;
	}

The idea is you’d pull the appropriate axis out of the camera matrix (depending if you wanted to turn up/right/tilt) and use that as an axis, build the matrix with the function above, transpose it and multiply it with the current camera matrix.

This isn’t the “best” way because over time you’d accumulate rounding errors that slightly distort your matrix (and you’d need to renormalize those axis vectors in the top left 3x3 of the camera matrix). A better way would be to represent your camera orientation using a quaternion (and keep a vector for its position) and build your camera matrix directly from that. It’s a lot cheaper to normalize a quaternion.