Shadowmap Memory Barrier

Vulkan
#1

I noticed my shadow map renders were sometimes experiencing a single frame lag where the old shadow map was shown in the scene render:

I started out to fix the memory barrier settings to try to eliminate this error, using this tutorial as a guide to the correct settings. This is a little different because it is using the 2KHR version of vkCommandPipelineBarrier(), which I have not used before:

In my code, following the shadow map rendering, I add a memory barrier like below:

//Image layout transitions
vector<VkImageMemoryBarrier2KHR> barriers;
VkImageMemoryBarrier2KHR barrier = {};
barrier.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2;
barrier.srcStageMask = VK_PIPELINE_STAGE_2_EARLY_FRAGMENT_TESTS_BIT_KHR | VK_PIPELINE_STAGE_2_LATE_FRAGMENT_TESTS_BIT_KHR;
barrier.srcAccessMask = VK_ACCESS_2_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT_KHR;
barrier.dstStageMask = VK_PIPELINE_STAGE_2_FRAGMENT_SHADER_BIT_KHR;
barrier.dstAccessMask = VK_ACCESS_2_SHADER_READ_BIT_KHR;		
barrier.newLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
barrier.subresourceRange.levelCount = 1;

for (auto pair : visset->cameravislists)
{
	auto light = pair.first->As<RenderLight>();
	if (light == NULL) continue;
	barrier.oldLayout = light->shadowbuffer[0]->depthtexture->GetLayout();
	barrier.subresourceRange.layerCount = 1;
	if (light->description.type == LIGHT_POINT and MULTIPASS_CUBEMAP == true) barrier.subresourceRange.layerCount = 6;
	barrier.image = light->shadowbuffer[0]->depthtexture->vkimage;
	barrier.subresourceRange.aspectMask = VK_IMAGE_ASPECT_DEPTH_BIT;
	switch (light->shadowbuffer[0]->depthtexture->format)
	{
	case VK_FORMAT_D24_UNORM_S8_UINT:
	case VK_FORMAT_D32_SFLOAT_S8_UINT:
	case VK_FORMAT_D16_UNORM_S8_UINT:
		barrier.subresourceRange.aspectMask |= VK_IMAGE_ASPECT_STENCIL_BIT;
	}
	barriers.push_back(barrier);
	light->shadowbuffer[0]->depthtexture->imagelayout[0][0] = barrier.newLayout;
	if (light->shadowcache[0])
	{
		barrier.image = light->shadowcache[0]->depthtexture->vkimage;
		barriers.push_back(barrier);
		light->shadowcache[0]->depthtexture->imagelayout[0][0] = barrier.newLayout;
	}
}

VkDependencyInfoKHR dependencyInfo = {};
dependencyInfo.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO_KHR;
dependencyInfo.imageMemoryBarrierCount = barriers.size();
dependencyInfo.pImageMemoryBarriers = barriers.data();
device->vkCmdPipelineBarrier2KHR(commandbuffers[currentFrame]->commandbuffer, &dependencyInfo);

I am using Vulkan 1.3. Validation layer VK_LAYER_KHRONOS_validation is enabled and it does not catch any errors.

I have no idea what is wrong with this?

1 Like
#2

I found this info:

I’ll give it a try tomorrow and see how it goes.

1 Like
#3

I have updated the original post with current information.

#4

I solved the crash by adding VK_KHR_synchronization2 to the list of required device extensions, but the original problem remains.

#5

You need to explicitly enable synchronization validation. It is off by default, because it could be expensive.

1 Like
#7

What is synchronization validation? Is this a validation layer? I tried adding “VK_LAYER_KHRONOS_synchronization2” to the validation layers but it had no effect.

#8

VkValidationFeatureEnableEXT::VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT

#9

Okay, I have enabled this feature in the createInstanceInfo structure:

#ifdef _DEBUG
		//Enable synchronization validation
		VkValidationFeaturesEXT validationfeatures = {};
		VkValidationFeatureEnableEXT validationfeaturesenable = VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT;
		validationfeatures.sType = VK_STRUCTURE_TYPE_VALIDATION_FEATURES_EXT;
		validationfeatures.enabledValidationFeatureCount = 1;
		validationfeatures.pEnabledValidationFeatures = &validationfeaturesenable;
		createInfo.pNext = &validationfeatures;
#endif

		res = vkCreateInstance(&createInfo, nullptr, &instance);

What is supposed to happen? My debug report callback used for validation layers is not being called.

#10
  • VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT specifies that Vulkan synchronization validation is enabled. This feature reports resource access conflicts due to missing or incorrect synchronization operations between actions (Draw, Copy, Dispatch, Blit) reading or writing the same regions of memory.

If it is not reporting anything, then you enabled validation wrong™, the layers are not complete enough to catch the problem, or it is not a synchronization problem.

You could try to violate something more blatantly to make sure your validation setup works correctly.

#11

I have been using validation a long time.

Maybe there is another flag needed here?:

VkDebugReportCallbackCreateInfoEXT callbackCreateInfo = {};
callbackCreateInfo.sType = VK_STRUCTURE_TYPE_DEBUG_REPORT_CREATE_INFO_EXT;
callbackCreateInfo.flags = VK_DEBUG_REPORT_ERROR_BIT_EXT | VK_DEBUG_REPORT_WARNING_BIT_EXT | VK_DEBUG_REPORT_PERFORMANCE_WARNING_BIT_EXT;
callbackCreateInfo.pfnCallback = &DebugReportCallback;
VkResult result = vkCreateDebugReportCallbackEXT(instance, &callbackCreateInfo, NULL, &callback);
VkAssert(result)
#12

My general validation error reporting is working correctly. I’ve also set up what I think is a correct barrier with the older API, but it does not alleviate the original problem:

vector<VkImageMemoryBarrier> barriers;
VkImageMemoryBarrier barrier = {};
barrier.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER;
barrier.srcAccessMask = VK_ACCESS_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT;
barrier.dstAccessMask = VK_ACCESS_SHADER_READ_BIT,
barrier.newLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
barrier.subresourceRange.baseMipLevel = 0;
barrier.subresourceRange.levelCount = 1;
barrier.subresourceRange.baseArrayLayer = 0;
barrier.subresourceRange.layerCount = 1;
for (auto pair : visset->cameravislists)
{
	auto light = pair.first->As<RenderLight>();
	if (light == NULL) continue;
	barrier.oldLayout = light->shadowbuffer[0]->depthtexture->GetLayout();
	light->shadowbuffer[0]->depthtexture->imagelayout[0][0] = barrier.newLayout;
	barrier.image = light->shadowbuffer[0]->depthtexture->vkimage;
	barrier.subresourceRange.aspectMask = VK_IMAGE_ASPECT_DEPTH_BIT;
	switch (light->shadowbuffer[0]->depthtexture->format)
	{
	case VK_FORMAT_D24_UNORM_S8_UINT:
	case VK_FORMAT_D32_SFLOAT_S8_UINT:
	case VK_FORMAT_D16_UNORM_S8_UINT:
		barrier.subresourceRange.aspectMask |= VK_IMAGE_ASPECT_STENCIL_BIT;
		break;
	}
	barriers.push_back(barrier);
}
if (!barriers.empty())
{
	vkCmdPipelineBarrier(
		commandbuffers[currentFrame]->commandbuffer,
		VK_PIPELINE_STAGE_EARLY_FRAGMENT_TESTS_BIT | VK_PIPELINE_STAGE_LATE_FRAGMENT_TESTS_BIT,
		VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT,
		0, 0, NULL, 0, NULL, barriers.size(), barriers.data());
}
#14

Maybe there is another flag needed here?

No, all it needs is add the VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT and boilerplate to pNext and enable the extension that defines this enum.

I’ve also set up what I think is a correct barrier with the older API, but it does not alleviate the original problem:

Problem with excerpts is the autor rarely shows the actual bug. Looks vaguely sane. Not much more that can be done here without complete minimal example.

Kinda sus it is a barrier and not render pass dependency.

Try more brutal synchronization (e.g. vk*WaitIdle) to eliminate if it even is sync problem. Make sure that the synchronization primitive is actually executed between the two things it is supposed to separate.

Also check if you are looking at the right thing. Looks like it is exactly one frame delayed. Might be a swapchain related problem, or bad management of multi-buffering of other resources.

1 Like
#15

Wait, what is this? I have never heard of such a command, and can find nothing on it…

#16

vkDeviceWaitIdle or vkQueueWaitIdle. I meant *, not sure why I have gone with X…

PS: Ups, also no Cmd.

Also less brutal option (and potentially less reliable for debugging purposes) is the total barrier. Anyway the debugging strategy here is to oversynchronize to check if we are even after the right problem.

1 Like
#17

Yeah, but that would get run as the command buffer is being recorded, not inside the command buffer.

I added this:

VkMemoryBarrier2KHR memoryBarrier = {};
memoryBarrier.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER_2;
memoryBarrier.srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT_KHR,
memoryBarrier.srcAccessMask = VK_ACCESS_2_MEMORY_READ_BIT_KHR |   VK_ACCESS_2_MEMORY_WRITE_BIT_KHR,
memoryBarrier.dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT_KHR,
memoryBarrier.dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT_KHR |   VK_ACCESS_2_MEMORY_WRITE_BIT_KHR;

VkDependencyInfoKHR dependencyInfo = {};
dependencyInfo.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO;
dependencyInfo.memoryBarrierCount = 1;
dependencyInfo.pMemoryBarriers = &memoryBarrier;

device->vkCmdPipelineBarrier2KHR(commandbuffers[currentFrame]->commandbuffer, &dependencyInfo);

And I can still see the error occurring occasionally.

Hold on, I need to do some more testing…

#18

Okay, the good news is I am a better Vulkan programmer than I thought, because my Vulkan code is flawless.

The bad news is I am worse at multithreading than I thought. :smiley:

What was happening is the light position was changing, but a new visibility list had not been received yet from the culling thread. So the light 4x4 matrix was moving but the shadow did not get updated until a signal was sent to the culling thread indicating that the light had shifted, and a new vis list was returned that included the visibility set for the light’s shadow render. The result was new light position with old out-of-date shadow map. My solution was to store an extra 4x4 matrix that only gets updated when the shadow is rendered, and use that in the scene render for the shadowmap lookup, which sounds kind of obvious now.

Basically.

Thank you for your help determining that memory barriers were not the issue.