OpenGL coordinate system to Vulkan (Perspective Matrix)

I’m having trouble getting my projection matrix to work in Vulkan. I’m using glm 0.9.7.4. (GLM_FORCE_DEPTH_ZERO_TO_ONE is not enabled.)

This is what my scene looks like when using the same projection matrix I’ve used for OpenGL:

Since OpenGL uses a left-handed system and Vulkan uses a right-handed system, this is the expected result.

The view and projection matrices for this scenes are set up like this:


glm::vec3 pos{1203.77f,-127.22f,141.378f};
glm::vec3 forward{0.908188f,-0.185572f,0.375178f};
glm::vec3 up{0.f,1.f,0.f};
auto v = glm::lookAt(pos,pos +forward,up);
auto p = glm::perspective(glm::radians(75.f),1024.f /768.f,1.f,32768.f);

[...]
auto mvp = p *v *m; // MVP is being sent to shader, no additional transformations are done

To get the right orientation, I scaled the projection matrix by -1 on the y-axis (And flipped the face winding order):


glm::vec3 pos{1203.77f,-127.22f,141.378f};
glm::vec3 forward{0.908188f,-0.185572f,0.375178f};
glm::vec3 up{0.f,1.f,0.f};
auto v = glm::lookAt(pos,pos +forward,up);
auto p = glm::perspective(glm::radians(75.f),1024.f /768.f,1.f,32768.f);
p = glm::scale(p,glm::vec3{1.f,-1.f,1.f});

This kind of works, however with an odd side-effect:
With front-face culling nothing is rendered at all.
With back-face culling about half of the objects are rendered:

With no culling enabled, all objects are rendered correctly:

This doesn’t really make any sense to make, and happens regardless of whether I’m using clockwise or counter-clockwise face winding order.

After some searching I’ve stumbled over this page which goes into a bit more detail about the differences between OpenGL and Vulkan. The matrix at the end of the article is supposed to do all of the correct conversions (Including depth), so I gave that a try:


glm::vec3 pos{1203.77f,-127.22f,141.378f};
glm::vec3 forward{0.908188f,-0.185572f,0.375178f};
glm::vec3 up{0.f,1.f,0.f};
auto v = glm::lookAt(pos,pos +forward,up);
auto p = glm::perspective(glm::radians(75.f),1024.f /768.f,1.f,32768.f);
const glm::mat4 clip{
	1.f,0.f,0.f,0.f,
	0.f,-1.f,0.f,0.f,
	0.f,0.f,0.5f,0.5f,
	0.f,0.f,0.f,1.f
};
p = clip *p;

Doing that results in the exact same culling issues (Somehow due to the inverted y-component I’m guessing), and incorrect fov:

Why does that affect my fov at all, and is there anything else I need to take into account?

Why not GLM_DEPTH_ZERO_TO_ONE? It fixes glm::perspective() to work with Vulkan.

And do you have GLM_FORCE_LEFT_HANDED set?

I figured since they use that matrix in the official demos, it’d be the safest solution.
Anyway, I’ve just tried it with GLM_DEPTH_ZERO_TO_ONE and scaling by vec3(1,-1,1) (Which should be the equivalent of what that matrix does?), and it ends up with the correct orientation and fov, but the face culling is still behaving oddly.
Here’s my current code:


// GLM_DEPTH_ZERO_TO_ONE is enabled
glm::vec3 pos{1203.77f,-127.22f,141.378f};
glm::vec3 forward{0.908188f,-0.185572f,0.375178f};
glm::vec3 up{0.f,1.f,0.f};
auto v = glm::lookAtRH(pos,pos +forward,up);
auto p = glm::perspectiveRH(glm::radians(75.f),1024.f /768.f,1.f,32768.f);
p = glm::scale(p,glm::vec3{1.f,-1.f,1.f});

VK_FRONT_FACE_COUNTER_CLOCKWISE +VK_CULL_MODE_BACK_BIT / VK_FRONT_FACE_CLOCKWISE +VK_CULL_MODE_FRONT_BIT: Nothing is being rendered
VK_FRONT_FACE_CLOCKWISE +VK_CULL_MODE_BACK_BIT / VK_FRONT_FACE_COUNTER_CLOCKWISE +VK_CULL_MODE_FRONT_BIT: Half of the objects are being rendered

VK_FRONT_FACE_CLOCKWISE +VK_CULL_MODE_NONE / VK_FRONT_FACE_COUNTER_CLOCKWISE +VK_CULL_MODE_NONE: All objects are being rendered

It’s not, the only glm define I have enabled now is “GLM_DEPTH_ZERO_TO_ONE”.

Well, they have the matrix transposed to yours (column/row major order mess?). That’s perhaps, why it messes your FOV in that case…

I also have another problem with shadow mapping, which I believe might be related.

Here are the matrices for my light source:


auto p = glm::perspectiveRH<float>(glm::radians(90.f),1.f,1.f,lightDistance);
auto v = glm::lookAtRH(GetPosition(),GetPosition() +m_direction,glm::vec3(0,1,0))
auto mv = p *v;

The mv matrix is used for rendering the actual shadows to the shadow map. This part works just fine.

When rendering my scene, I’m using the same matrix for the light, multiplied with this bias matrix so I can actually sample from the shadow map in [0,1]:


const glm::mat4 bias(
	0.5f,0.0f,0.0f,0.0f,
	0.0f,0.5f,0.0f,0.0f,
	0.0f,0.0f,0.5f,0.0f,
	0.5f,0.5f,0.5f,1.0f
);
mv = bias *mv;

This is then used to transform the vertex position to light space in the vertex shader:


vec4 vertex_pos_lightSpace = lightSource.depthVP *ModelMatrix *vec4(vertexPosition,1.0); // lightSource.depthVP = bias *mv (Same as above)

In the fragment shader, I then just grab the sample from the shadow map (The result is multiplied with the light/diffuse color):


layout(set = 2,binding = 0) uniform sampler2DShadow shadowMap;
[...]
float calculate_shadow_term(sampler2DShadow shadowMap,vec4 shadowCoord,float bias) // shadowCoord = vertex_pos_lightSpace
{
	shadowCoord.w += bias;
	return textureProj(shadowMap,vec4(
		shadowCoord.x +u_renderSettings.shadowRatioX *shadowCoord.w, // u_renderSettings.shadowRatioX is 1.0
		shadowCoord.y +u_renderSettings.shadowRatioY *shadowCoord.w, // u_renderSettings.shadowRatioY is 1.0
		shadowCoord.z,shadowCoord.w
	),bias);
}

However, the function just returns 0 for all fragments, resulting in an unlit scene:

You can see the contents of the shadow map at the top left (Click the image for a larger version). The depth values in that image are linearized to make them more clearly visible (They’re not linearized for the actual shadow map).

So, then I’ve tried Sascha Willems’ version for the depth lookup to see if that’s any different (Essentially the same as textureProj, but done by hand):


layout(set = 2,binding = 0) uniform sampler2D shadowMap;
[...]
float calculate_shadow_term(sampler2D shadowMap,vec4 shadowCoord,float bias)
{
	shadowCoord.w += bias;

	float shadow = 1.0;
	vec4 P = shadowCoord /shadowCoord.w;
	shadowCoord = P /P.w;
	if(shadowCoord.z > 0.0 && shadowCoord.z < 1.0)
	{
		float dist = texture(shadowMap,shadowCoord.st).r +0.0025;
		if(shadowCoord.w > 0.0 && dist < shadowCoord.z) 
			return 0.2; // Ambient
	}
	return shadow;
}

Interesting to note is that if I don’t add the offset ‘0.0025’, everything is in shadow:

(Just the ambient light coming through)

However, with the offset, I can see the shadows (More or less);

So the actual sampling is correct, but it looks like the sampled depth value from the shadow map is incompatible with the depth value from the scene (Which is probably why textureProj fails). I’m using the same shadow matrix on both shaders (shadow shader and scene shader). Plus, the same setup worked in OpenGL just fine, so my guess is that it’s something to do with the differences in Vulkan when handling depth values.
I’ve set the depth range for both shaders to [0,1].

What could be the cause of this?