This advice is off the top of my head (literally, I found it written there…) so if it is wrong then shoot someone…
When you rotate the person you need to get the reverse of the viewing vector (which would be rotated by same matrix you used to rotate the person). Then get the centre point of the person and translate that value, using the reverse viewing vector, by the distance of the camera. You now have a point to rotate (slerp) to…
ALTERNATIVELY,
using gluLookAt, set the viewing point to that of the person, then simply rotate the camera by the same amount you rotated the person…