Rotate an object around a fixed point calculated from mouse position

OpenGL OpenGL: Basic Coding
1

Hi, I’m implementing a rotating feature for my point cloud rendering application. I want to rotate the model around my mouse click point.

I tried to follow the order: Translate(x,y,z) → Rotate(angleX, angleY, 0) → Translate(-x,-y,-z).

PROBLEM:
After I rotate my model, I click on different position to calculate the fixed point, my model teleported to another place on my screen. I think is because I rotate the whole world’s coordinate, so the model keeps teleporting to another place.

Here is my render code:

GL.MatrixMode(MatrixMode.Modelview);
GL.LoadIdentity();
GL.Translate(transX, transY, 0);

GL.PushMatrix();
if (rotatePoint != null)
{
    // Draw clicked point
    GL.PointSize(30);
    GL.Begin(PrimitiveType.Points);
    GL.Color3(1.0, 0.0, 0.0);
    GL.Vertex3(rotatePoint.point.X, rotatePoint.point.Y, rotatePoint.point.Z);
    GL.End();

    // Translate to the rotatePoint
    GL.Translate(rotatePoint.point.X, rotatePoint.point.Y, rotatePoint.point.Z);
    
    GL.Rotate(angleY, 1, 0, 0);
    GL.Rotate(-angleX, 0, 1, 0);

    // Translate back
    GL.Translate(-rotatePoint.point.X, -rotatePoint.point.Y, -rotatePoint.point.Z);
}

SetupViewport();
pco.Render(point_size, ShowOctreeOutline, PointCloudColor, mFrustum);
GL.PopMatrix();
2

Here is the video describe my problem
(https://www.youtube.com/watch?v=YCDXockJydM)

3

In my code, I didn’t save the previous state and then multiply that previous state to the transformation I did earlier.

The problem is I keep creating a new model matrix base on the origin state so that make my model shifts when I choose a new pivot base on the NEW STATE but meanwhile reset the state to origin. (Example code):

// Global variable
Matrix4d prevModelMatrix = Matrix4d.Identity;
Matrix4d modelMatrix = Matrix4d.Identity;

function Render() {
  ...
  GL.LoadMatrix(modelMatrix);
  ...
  Transformation... (Use offset instead of using new rotate and translate value to avoid accumulating)
  For example: 
    - GL.Rotate(offsetAngleX, 1,0,0);
}

// Reset offsets to 0 to avoid Render() function still use the offset to transform the scene
function MouseUp() {
  offsetAngleX = 0;
  ...
}