I have been trying to solve this problem for a couple weeks now and I need some help. The problem I am having regardless of what system I use (Euler, Axis-Angle, Quaternion) is how to determine the appropriate future rotation from the changing local axis. For example:

Camera is at position (0,0,0) for simplicity. OUT is +z, UP is +y, RIGHT is +x. Rotation of the camera using arrow keys is relative to the screen at all times. As soon as I rotate I will not be aligned with the global axis anymore, so I have to assign x, y, and z rotation accordingly.

The confusion is something like this: I know for quaternions you can multiply Q1 * Q2 for example where Q1 is your current orientation and Q2 is a change to that orientation. So if I press the up arrow, how do I create Q2 to always be a rotation towards the top of the screen? The same issue applies to axis-angle format as well.

Do I need to store and manipulate the vector for each axis for each rotation or is there a more direct approach (can you “lose” your up vector or do all 4D vectors preserve the orientation)?

Isn’t the point of quaternions that you can do movement in multiple axis at the same time and arrive at the same orientation regardless of the order in which you applied them? Euler angles affect each other and cause gimbal lock due to their interaction.

The easiest type for me to visualize is axis-angle, but I don’t know how to do relative rotations for that as each axis changes. I imagine tracing the inside of a sphere where the XYZ tells me where I am and the angle tells me which direction is the relative “up” for example. It’s necessary to do all the rotations from the local perspective, but how?

I guess the issue is with all the things I’ve been reading, I know how to convert between the different types, to some degree how to multiply the current orientation by an offset, but not how to derive what I need to multiply into the current orientation based on my goals.

A quick and dirty way of doing this is to use a matrix for your rotations. Basically a 3x3 rotation matrix (or the top 3x3 of a 4x4 matrix) is the direction vectors right, forward and up. (assuming no scaling)

Isn’t the point of quaternions that you can do movement in multiple axis at the same time and arrive at the same orientation regardless of the order in which you applied them?

That’s incorrect. Rotations in 3D space are non-commutative transformations.

You don’t need to keep local axes, because they can be easily reconstructed at any time.
Assuming Q(V) = (Q-1)VQ = transformed vertex V by quaternion Q
Assuming you want to rotate around local X by w angle, and the original orientation is Q0:

X = Q0(V(1,0,0)) - axis vector in parent coordinate system (rotation only)
Q1 = Q0 * Q(X,w) - new orientation

or I need to do something like this: convert current angles and delta of the angles into matrices, multiple them, then extract angles from matrix and release one. I think a complex matrix XYZT is faster than XYZ*T one by one