Math: transforming a vector

[john_connor]

hi there,

i want to rotate/transform a vector, but i dont get it quite right. i’m using glm.

here’s my c++ code:

glm::vec3 Collision::Polytope::FurthestPoint(
	const glm::mat4& transform,
	const glm::vec3& direction) const
{
	if (Points.empty())
	{
		cout << "error Collision::Polytope::FurthestPoint(): polytope has no points" << endl;
		return { 0, 0, 0 };
	}

	vec3 p0 = transform * vec4(Points.at(0), 1);
	float max_distance = dot(p0, direction);
	vec3 furthest_point = p0;
	
	for (auto& point : Points)
	{
		vec3 p = transform * vec4(point, 1);
		float distance = dot(p, direction);
		if (distance < max_distance)
			continue;
	
		max_distance = distance;
		furthest_point = p;
	}

	return furthest_point;
}

what i’m doing here is:
i transform all points of a (convex) polytome (or mesh) using a “mesh-space to world-space” transformation (just translation + rotation, NO scaling). then i’m figuring out whith point lies furthest in a certain direction (in world-space).

what i want to do is to transform the direction, not each vertex (because it might be faster that way).

replacing “vec3 direction” with “inverse(transform) * vec4(direction, 0)” doesnt work.

what am i missing ?? thanks in advance!

[GClements]

You need the transpose, not the inverse:

a·b = a^Tb
=> (ma)·b = (ma)^Tb
= (a^Tm^T)b
= a^T(m^Tb)
= a·(m^Tb)

Also: transform is affine (just rotation, scale and translation, no perspective projection), right?

[john_connor]

ok, i’ll try the transpose instead … thank you!
yes, there’s no perspective projection applied, no camera space either