Vulkan : hlsl is not working with glslang. possibly a glslang issue? (EDIT : same problem with DXC)

Hey !
I’m working on a vulkan game engine where my shaders are compiled from HLSL or GLSL using glslang.

Problem : the spirv generated from hlsl code seems to be broken : it doesn’t write to the framebuffers, and there are some weird behaviors when I use the RenderDoc mesh viewer :

=> With a HLSL shader, RenderDoc seems to fail to determine the view’s frustum (the left pannel is grayed out)

The render pipeline is exactly the same, the code is equivalent. The only difference is the shader language. Am i missing something or should I post an issue on the glslang github ?
(This is the first time i use HLSL with vulkan so i maybe missed something obvious)

This is the glsl code (working) :

#pragma shader_language GLSL

// NOTE : this file is not directly sent to the compiller, 
// the "global" and "vertex" blocks are merged into the vertex shader code, 
// and the "global" and "fragment" blocks are merged into the fragment shader code.

global(gbuffer)
[
	#version 450
]

vertex(gbuffer)
[
	layout(location = 0) in vec3 inPos;

	layout(binding = 0) uniform UBO 
	{
		mat4 projection;
		mat4 view;
	} view_ubo;

	void main()
	{
		vec2 uv = vec2((gl_VertexIndex << 1) & 2, gl_VertexIndex & 2);
		gl_Position = vec4(uv * 2.0f - 1.0f, 0.0f, 1.0f);
		//view_ubo.projection * view_ubo.view * vec4(inPos, 0);
	}
]

fragment(gbuffer)
[
	layout (location = 0) out vec4 outPosition;
	layout (location = 1) out vec4 outNormal;
	layout (location = 2) out vec4 outAlbedo;

	void main()
	{
		outPosition = vec4(1, 1, 1, 1.0);
		outNormal = vec4(1, 1, 1, 1.0);
		outAlbedo = vec4(1, 1, 1, 1.0);
	}
]

and the hlsl version (broken)

#pragma shader_language HLSL

// NOTE : this file is not directly sent to the compiller, 
// the "global" and "vertex" blocks are merged into the vertex shader code, 
// and the "global" and "fragment" blocks are merged into the fragment shader code.

global(gbuffer)
[
	int _Subdivision = 4;
	int _Width = 1;

	struct VsToFsStruct
	{
		[[vk::location(0)]] float4 Pos : POSITION0;
	};
]

vertex(gbuffer)
[
	struct ViewUbo
	{
		float4x4 projection;
		float4x4 view;
		float end;
	};

	cbuffer view_ubo : register(b0)
	{
		ViewUbo view_ubo;
	}

	struct VsInput
	{
		float3 pos : POSITION0;
		uint vertex_index : SV_VertexID;
	};

	VsToFsStruct main(VsInput input)
	{
		VsToFsStruct output = (VsToFsStruct) 0;

		float2 uv = float2((input.vertex_index << 1) & 2, input.vertex_index & 2);
		output.Pos = float4(uv * 2.0f - 1.0f, 0.0f, 1.0f);
		return output;
	}
];

fragment(gbuffer) 
[
	struct FSOutput
	{
		float4 Color1 : SV_TARGET0;
		float4 Color2 : SV_TARGET1;
		float4 Color3 : SV_TARGET2;
	};
		
	FSOutput main(VsToFsStruct input)
	{
		FSOutput output = (FSOutput) 0;
		output.Color1 = float4(1, 10, 5, 1);
		output.Color2 = float4(1, 1, 1, 1);
		output.Color3 = float4(1, 1, 1, 1);
		return output;
	}
];

The function that generate the TShader :

https://github.com/PierreEVEN/HeadlessEngine/blob/2d97b3d6a87d3cd3b9ba88afe47a16a56315ffd6/src/engine/shader_builder/private/compiler.cpp#L305

And the TProgram : https://github.com/PierreEVEN/HeadlessEngine/blob/2d97b3d6a87d3cd3b9ba88afe47a16a56315ffd6/src/engine/shader_builder/private/compiler.cpp#L263

EDIT : It is not a glslang problem, I added a directX Shader Compiler backend, and I still have the same output :confused:

You should post the actual HLSL that gets sent to the compiler.

Also the 2 shaders are not the same. outPosition in your GLSL is assigned a different value to output.Color1 in your HLSL.

I’ve got the problem (yep I should have posted the precompiled version, but it wasn’t that) !
All the HLSL examples I had were written using an old shader model (VulkanExamples for example). While I was using shader model 6.

So the fix is quite dumb :

		float3 pos : POSITION0;

becomes

		float3 pos : Sv_Position;

I really suspected this kind of problem, but only having examples that tell me it should works convinced me it was something else.

Finally, by reading the generated spirv, someone noticed that I had an output problem. (not so obvious)

Well I’ll know for the next time render doc fail to find the view’s frustum. :slight_smile: