How to Rotate Camera in 3d with 2D point

Dear All
I am trying to 3d rotate the Camera Position from Camera Target using Mouse.
the camera rotates in orbit like in 3ds Max
I don’t know how to make it. so I google it and find many examples but not as I am looking for and I just mix with many codes with a lot of tries.
here is my code

void x_GL_Document::OnMouseMove(POINT pt)
{
	if (m_IsLBtnDown == true)
	{
		m_ActiveCam->Rotate(pt.x - m_ClickPt.x, pt.y -m_ClickPt.y);
		m_ClickPt = pt;
	}
}

void x_Camera::Rotate(int x, int y) // x and y is from mouse move point
{
	auto speed = 0.01f;
	auto rx = glm::radians(_ToFloat(x));
	auto ry = glm::radians(_ToFloat(y));

	m_CamRotate.x = cos(ry * cos(rx));
	m_CamRotate.y = sin(rx);
	m_CamRotate.z = sin(ry) * cos(rx);
	
	m_Front = glm::normalize(m_CamRotate);
	m_Up = glm::rotate(m_Up, m_CamRotate.z, m_Front);
	m_Position = glm::rotate(m_Position, m_CamRotate.x * speed, m_Up);
	m_Position = glm::rotate(m_Position, m_CamRotate.y * speed, m_Up);
	m_Position = glm::rotate(m_Position, m_CamRotate.z * speed, m_Up);
}

it would be great if someone helps me to correct it.
thanks in advance.

you’r sure that you don’t mean that you’r trying to change the direction of the camera, and not it’s position? The latter is usually done by a translation.
If the camera is not positioned in (0,0,0) you’ll probably not get the result that you expect.
You are dissatisfied with the outcome of your action … does it compile at all? I cannot find a glm::rotate() that takes a vector as first argument … are you missing a rotation-matrix?
transformations

hi
well, I am trying to change (rotate) the camera position. the position is Vec3. eg position = vec3(-20,-20,100). and target at vec3(0,0,0).

I am trying to rotate the camera position in a spherical way from the Camera Target. as per (int x, int y)
the x and y might be 0 or +1 or -1 only. as you can see it comes from mouse move.because when we change camera position we can see visually. it is moving or not. the Direction of the Camera I think it is From Camera Position to Camera Target. so when the camera position change it will change direction also.

yes there is glm::rotate function

    	    template<typename T, qualifier Q>
        	GLM_FUNC_QUALIFIER vec<3, T, Q> rotate
        	(
        		vec<3, T, Q> const& v,
        		T const& angle,
        		vec<3, T, Q> const& normal
        	)
        	{
        		return mat<3, 3, T, Q>(glm::rotate(angle, normal)) * v;
        	}

I want to like Song Ho Ahn Example. when I check his code. he has just use matrix only. did not get how to change the camera position with it.

here is his code

     // tramsform camera
    matrixView.identity();
    matrixView.rotateY(cameraAngleY);
    matrixView.rotateX(cameraAngleX);
    matrixView.translate(0, 0, -cameraDistance);
    glLoadMatrixf(matrixView.get());

ok, this sounds like:
glm::lookAt (vec< 3, T, Q > const &eye, vec< 3, T, Q > const &center, vec< 3, T, Q > const &up)

That leaves the calculation of the new position of the camera after drag. I would probably read dX and dY straight out as converted to angles. I did implement this once. As a user I preferred a clean vertical or horizontal move, so I chose vertical/horizontal on the first values of dX & dY (whichever was largest) and only used that direction on the current drag. But, ok, you may be a pilot … that’ll leave some angles to keep track of.
You havn’t yet told us what your outcome is…

@CarstenT

it rotates the camera. but not as it needs to rotate. here is my camera class. Rotate and Pan not working perfectly.
edit: I am using modern OpenGL.

 class x_Camera
{
public:
	glm::vec3 m_Position{0,0,30};
	glm::vec3 m_TargetAt = {0,0,0};
	glm::vec3 m_Up = { 0.0f,1.0f,0.0f };
	float m_Near = 0.0111f;
	float m_Far = 1000.0f;
	float m_ZoomScale = 1.1f;
	glm::vec3 m_CamRotate = {0.0f,0.0f,0.0f};
	glm::mat4 Get_MVP_Matrix();
	glm::mat4 Get_View_Matrix();
	glm::mat4 Get_Model_Matrix();
	glm::mat4 Get_Projection_Matrix();
	glm::vec3 m_Front;
	float _fAspect = 1.0f;
	float m_FOV = 45;
	bool m_Orthographic = true;
	void Rotate(int x, int y);
	float GetDistance();
	void Pan(POINT pt);
	glm::vec3 lookAtToAngles(const glm::vec3 pos, const glm::vec3 lookat);
	glm::vec3 GetDirection();
	glm::vec3 GetRight();
private:
	glm::mat4 m_Model = glm::mat4(1.0f), m_View = glm::mat4(1.0f), m_Projection = glm::mat4(1.0f);
};

glm::mat4 x_Camera::Get_MVP_Matrix()
{
	glm::mat4 MVP;
	if (m_Orthographic == false)
	{
		m_Projection = glm::perspective(glm::radians(m_FOV) * m_ZoomScale, _fAspect, this->m_Near, this->m_Far);
	}
	else
	{
		m_Projection = glm::ortho(-1.0f * m_ZoomScale, 1.0f * m_ZoomScale, -1.0f * m_ZoomScale, 1.0f * m_ZoomScale, this->m_Near, this->m_Far);
	}
	m_View = glm::lookAt(this->m_Position, this->m_TargetAt, this->m_Up);

	MVP = m_Projection * m_View * m_Model;
	return MVP;
}
glm::mat4 x_Camera::Get_View_Matrix()
{
	//glm::mat4 MVP;
	//glm::mat4 projection;
	if (m_Orthographic == false)
	{
		m_Projection = glm::perspective(glm::radians(m_FOV) * m_ZoomScale, _fAspect, this->m_Near, this->m_Far);
	}
	else
	{
		//auto t = glm::radians(m_FOV) * 10;
		m_Projection = glm::ortho(-1.0f * m_ZoomScale, 1.0f * m_ZoomScale, -1.0f * m_ZoomScale, 1.0f * m_ZoomScale, this->m_Near, this->m_Far);
	}
	return glm::lookAt(this->m_Position, this->m_TargetAt, this->m_Up);
}
glm::mat4 x_Camera::Get_Model_Matrix()
{
	return m_Model;
}
glm::mat4 x_Camera::Get_Projection_Matrix()
{
	if (m_Orthographic == false)
	{
		return glm::perspective(glm::radians(m_FOV) * m_ZoomScale, _fAspect, this->m_Near, this->m_Far);
	}
	else
	{
		//auto t = glm::radians(m_FOV) * 10;
		return glm::ortho(-1.0f * m_ZoomScale, 1.0f * m_ZoomScale, -1.0f * m_ZoomScale, 1.0f * m_ZoomScale, this->m_Near, this->m_Far);
	}
}
void x_Camera::Rotate(int x, int y)
{
	auto speed = 0.01f;
	auto rx = glm::radians(_ToFloat(x));
	auto ry = glm::radians(_ToFloat(y));

	m_CamRotate.x = cos(ry * cos(rx));
	m_CamRotate.y = sin(rx);
	m_CamRotate.z = sin(ry) * cos(rx);


	m_Model = glm::mat4(1.0f);
	//auto mDist = GetDistance();
	//m_Model = glm::rotate(m_Model, m_CamRotate.y, glm::vec3(0,1,0));
	//m_Model = glm::rotate(m_Model, m_CamRotate.x, glm::vec3(1, 0, 0));

	//m_Model = glm::translate(m_Model, glm::vec3(0, 0, -mDist));


	m_Front = glm::normalize(m_CamRotate);
	m_Up = glm::rotate(m_Up, m_CamRotate.z, m_Front);
	m_Position =  glm::rotate(m_Model, m_CamRotate.x * speed, m_Up) * glm::vec4(m_Position, 1.0f);
	m_Position =  glm::rotate(m_Model, m_CamRotate.y * speed, m_Up) * glm::vec4(m_Position, 1.0f);
	m_Position =  glm::rotate(m_Model, m_CamRotate.z * speed, m_Up) * glm::vec4(m_Position, 1.0f);
	//m_Position = glm::vec4(m_Position, 1.0f) * glm::rotate(m_Model, m_CamRotate.z * speed, glm::vec3(0, 0, 1));
	//m_Position = glm::vec4(m_Position, 1.0f) * glm::rotate(m_Model, m_CamRotate.x * speed, glm::vec3(0,1,0));
	//m_Position = glm::vec4(m_Position, 1.0f) * glm::rotate(m_Model, m_CamRotate.y * speed, glm::vec3(1, 0, 0));


}
float x_Camera::GetDistance()
{
	return glm::distance(m_Position, m_TargetAt);
}
void x_Camera::Pan(POINT pt) // not working perfect
{
	const float m_speed = 0.5f / m_ZoomScale;

	glm::vec3 panX = GetRight() * (pt.x * 1.0f);
	glm::vec3 panY = m_Up * (pt.y * 1.0f);
	m_Position -= (panX + panY) * m_speed;
	m_TargetAt -= (panX + panY) * m_speed;

	//m_Position = m_Position - glm::vec3(pt.x* m_speed,pt.y* m_speed, 0.0f);
	//m_TargetAt = m_TargetAt - glm::vec3(pt.x* m_speed, pt.y* m_speed, 0.0f);
}
glm::vec3 x_Camera::lookAtToAngles(const glm::vec3 pos, const glm::vec3 lookat)
{
	const float DEG2RAD = acos(-1) / 180.0f;  // PI/180
	const float RAD2DEG = 180.0f / 3.141592f;
	glm::vec3 vec;
	float  yaw, pitch;                          // roll is 0

	// compute the vector from origin point to lookat point
	vec = lookat - pos;

	if (vec.x == 0.0f && vec.y == 0.0f)          // vector is on the Y-axis, therefore,
	{                                           // Yaw is 0, and Pitch will be +90 or -90.
		yaw = 0.0f;
		if (vec.y >= 0.0f)
			pitch = 90.0f;                      // facing along +Y
		else
			pitch = -90.0f;                     // facing along -Y
	}
	else
	{
		// yaw: angle on X-Z plane (heading)
		// yaw should be 0 if facing along +Z initially
		yaw = RAD2DEG * atan2f(-vec.x, vec.z); // range -pi ~ +pi

		// length of vector projected on X-Z plane
		float dxz = sqrtf(vec.x * vec.x + vec.z * vec.z);
		pitch = RAD2DEG * atan2f(vec.y, dxz);  // range -pi ~ +pi
	}

	// store angles(degree) in the array
	return glm::vec3(pitch, yaw, 0);
}

glm::vec3 x_Camera::GetDirection()
{
	return  glm::normalize(m_Position - m_TargetAt);
}

glm::vec3 x_Camera::GetRight()
{
	return  glm::normalize(glm::cross(m_Up, GetDirection()));
}

roll is 0 is a clever decition … working with only two angle-variables suffice to uniquely identify a point on a sphere. Adding a third variable confuses things.
You can think of a matrix as a position and a direction, sometimes adressed as a pose. That makes a matrix a prime thingie for a model and a camera though you cannot right away use them as proj x camera x model, but they will store the info you need. So, you can learn to ‘read’ that info out of their matrices or decide to keep track of positions&directions for both. The latter is somewhat redundant since you must erect the matrices.
Model & camera-matrix refer to the same world-coordinates, the view_model matrix can be constructed from them.
You have worked a lot with the problem … maybe it’s time to do some clean-up. The class contains ‘everything’. You should separate model and projection-stuff out of it. A camera pose (matrix) could be a private member and the model_view matrix the desired output (taking the model-matrix as an argument). Having all the other variables floating around creats more confusion than help.
Give me a hint if you need support to ‘read’ the matrix.
… edit
it’s not you. It’s just difficult. But doable.
understanding-the-view-matrix