Hey, I noticed this thread because of stoo’s link to my web page.
I would recommend not to follow Q’s advice too closely. There is definitely no reason to store Euler angles as your default rotation representation. Frenet frames are a better idea but they are bulkier. Quaternions are probably your best bet.
For a given rotation, the quaternion, Euler angle, and Frenet frame store exactly the same thing. There’s no difference between how much information is present in each of these. e.g. it’s totally untrue that a quaternion has no notion of where “up” is and Euler angles somehow do.
The problem CatAtWork’s having is entirely in rotateTo. The point of this function is to find the smallest rotation between two headings. (There are always an infinite number of rotations that map one vector onto another one; you want the shortest one, usually). Well, when you take the case of a vector mapping to another vector that points in the opposite direction, suddenly every direction you rotate is equally as good – you can rotate around any axis, the answer is always that a 180 degree rotation will give you the other vector.
So the problem isn’t in the quaternion (or any kind of rotation representation). The problem is in the problem statement itself – trying to come up with the shortest-path rotation when there isn’t specifically such a thing.
The code that Q pasted in handles this case by tweaking around with the coordinates to just pull an axis out of thin air. This is usually how the problem gets solved (though if you look around you can find examples with code that is made to run really fast. My articles that stoo pointed to have one or two examples of that, but there are plenty of others).
The up-vector flip that CatAtWork sees is because rotateTo is handling this problem by pulling an axis out of thin air. This axis is discontinuous with what came before, so the resulting frame is oriented around the wrong way. There’s no way for rotateTo to do a better job of this for you, unless you had a version where you provide it more information (like some way to choose a preferred frame).
Given that, the solution you have (to fix the up-vector after the fact) is not too bad. You could also just write your own version of rotateTo that takes augmented information.
Also, just to make clear, the problem has absolutely nothing to do with gimbal lock (as madmortigan suggests). Gimbal lock does not happen when you use quaternions; that’s part of the point.