I need to align an object to another in 3d space. Each of them have three points lying in one plane (a center point and couple more to form a certain angle with the center point). I need the three points of the second object to overlap those of the first.

The translation parameters were simple… which is the difference between the two center points.

The rotational part is tricky… what I tried to do is that I got the difference in the planes’ normals’ angles in the x,y,z direction. When I try to apply such rotation to the second object, it never overlaps with the first one…

Since I couldn’t get that link to work, here’s a quick answer.

We want translations S and T and a rotation R such that doing (S then R then T) gives the required result, mapping points a1, a2, a3 onto b1, b2, b3 respectively in the correct way. (a1 and b1 are the “centre” points you mention). R will rotate about the origin, so S needs to take a1 to the origin, and T needs to take the origin to b1. With a little thought we get nine equations, written as three vector equations like so:

b1 = R (a1 - a1) + b1,
b2 = R (a2 - a1) + b1,
b3 = R (a3 - a1) + b1.

The first of the vector equations is vacuous. Let B2 = b2 - b1, B3 = b3 - b1 and A2 = a2 - a1, A3 = a3 - a1, so that

B2 = R A2,
B3 = R A3.

This leaves 6 equations for the 9 unknowns of R. We need a third independent vector equation to determine R. The obvious candidate is

B1 = R A1,

where A1 = A2 x A3 and B1 = B1 x B3. (That’s the cross product or vector product.)
Now write the three vector equations as a single matrix equation,

B = R A,

where the columns of B are B1, B2 and B3 and those of A are A1, A2 and A3. Solving the equation (which will certainly be possible provided the angle ‘a2-a1-a3’ is not 0 or 180 degrees, so that A2 and A3 are linearly independent) we get

R = B Ai

where Ai is the inverse of A. Under the conditions you described (the appropriate lengths and angles are equal), R is a rotation matrix.