How exactly does the up vector affect the camera? I know it tilts it to the side but is that all it does?
Yes, that’s all it does. A direction only (which you get form the eye and target position) is not enough to get a unique transformation. You need something to determine the roll about the direction vector, and that’s what the up vector is for.