I am drawing simple objects on the screen(teapot, circle). What I want to do is: when I select an object I want the camera to target the selected object smoothly.

At the moment I am able to target the selected object but without the animation. I would like some information of how to achieve the animation.

To manage my camera I use this struct:
struct raaCamera
{
float m_fVP[4];
float m_fVT[4];
float m_fVR[4];
float m_fVU[4];
float m_fVD[4];
};

Also I use my own maths for vector operations such as (add, subtract, multiply, projection …)

To manage the targeting at the moment I use this:
vecSet(x, y, z, g_Camera.m_fVT);
vecSub(g_Camera.m_fVT,g_Camera.m_fVP,g_Camera.m_fVD);
vecNormalise(g_Camera.m_fVD,g_Camera.m_fVD);

I implement camera rotation on mouse movement. The “step” is based on the mouse movement. I believe I can use a similar logic for this problem, but I can’t calculate the “step” that I need to target the selected object.

I can only suggest to check out transformation matrices, specifically 4x4 matrices used in OpenGL. Essentially it stores the same stuff you have in your structure, and more.

A 4x4 OpenGL-style matrix stores the right/up/forward vector in the first three columns, and the translation in the 4th. Off the top of my head, it should be something like this:

Rx Ux Fx Tx
Ry Uy Fy Ty
Rz Uz Fz Tz
0 0 0 1

Where R = unit vector pointing to right
Where U = unit vector pointing up (up-vector)
Where F = unit vector pointing to front (or rear, depending on your app’s convention)
Where T = translation vector (i.e. world space position coordinate)

The first three vectors together form the 3x3 sub-matrix to the top-left, which not coincidentally, also serves as rotational matrix.

Now as for transitioning between two transforms, you can simply do linear interpolation (LERP), but you have to renormalize the matrix. This involves fairly complex math and is somewhat computationally expensive. Though there are plenty of matrix libraries you can find on the web, many using the OpenGL convention. Many of them provide functions to get and set rotations in Euler format.

Then there’s the a problem called ‘gimbal lock’ (see wikipedia for details). If this is a problem for your application, you might want to look into quaternions, as these provide some alternative ways to do rotational transitions (with some very elegant math). Keyword: SLERP.