Multiple primary command buffers and VK_KHR_dynamic_rendering

[wookie]

Hello.

I’m currently looking into the VK_KHR_dynamic_rendering extension and I don’t really understand what does the specification mean with this:

Dynamic render passes can also span across multiple primary command buffers, rather than relying on secondary command buffers.

I get that the whole thing with having a render pass per command buffer goes away with this extension, but I don’t understand how can I span the dynamic render pass across multiple primary command buffers, as when calling vkCmdBeginRendering I still need to pass a handle to a command buffer.

Say I want to thread the recording of commands in some pass, without dynamic rendering, I could record the command into secondary command buffers on multiple cores. How would I achive something like this with dynamic rendering?
I assume that instead of having secondary command buffers, I’d allocate primary command buffers and submit them together, calling vkCmdBegingRendering with the same VkRenderingInfo.
Am I right or am I missing something?

[krOoze]

Previously vkCmdBeginRenderPass and vkCmdEndRenderpass must be entirely contained within one command buffer.

With Dynamic Rendering, you can suspend render pass instance with VK_RENDERING_SUSPENDING_BIT and resume it with VK_RENDERING_RESUMING_BIT.

[wookie]

Oh, now I see. Idk how I missed these flags, thanks!
So basically I can still do the thing with secondary command buffers that I mentioned, using VK_RENDERING_CONTENTS_SECONDARY_COMMAND_BUFFERS_BIT_KHR or suspend/resume the rendering with pipeline barriers when recording to multiple primary command buffers, right?

[wookie]

So I just tried the VK_RENDERING_CONTENTS_SECONDARY_COMMAND_BUFFERS_BIT approach.
I’ve setup both VkCommandBufferInheritanceRenderingInfo and VkCommandBufferInheritanceInfo, set VkCommandBufferInheritanceInfo::pNext to VkCommandBufferInheritanceRenderingInfo VkCommandBufferBeginInfo::pInheritanceInfo to VkCommandBufferInheritanceInfo.

I also have VkRenderingInfo::flags set to VK_RENDERING_CONTENTS_SECONDARY_COMMAND_BUFFERS_BIT.

That’s the setup:

        VkRenderingAttachmentInfo colorAttachmentInfo{};
        colorAttachmentInfo.sType              = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO;
        colorAttachmentInfo.pNext              = nullptr;
        colorAttachmentInfo.imageView          = GState.ColorImageView;
        colorAttachmentInfo.imageLayout        = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;
        colorAttachmentInfo.resolveMode        = VK_RESOLVE_MODE_AVERAGE_BIT;
        colorAttachmentInfo.resolveImageView   = GState.SwapChainImageViews[i];
        colorAttachmentInfo.resolveImageLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;
        colorAttachmentInfo.loadOp             = VK_ATTACHMENT_LOAD_OP_CLEAR;
        colorAttachmentInfo.storeOp            = VK_ATTACHMENT_STORE_OP_DONT_CARE;
        colorAttachmentInfo.clearValue.color   = { 0, 0, 0, 1 };

        VkRenderingAttachmentInfo depthStencilAttachmentInfo{};
        depthStencilAttachmentInfo.sType                   = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO;
        depthStencilAttachmentInfo.pNext                   = nullptr;
        depthStencilAttachmentInfo.imageView               = GState.DepthBufferImageView;
        depthStencilAttachmentInfo.imageLayout             = VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;
        depthStencilAttachmentInfo.loadOp                  = VK_ATTACHMENT_LOAD_OP_CLEAR;
        depthStencilAttachmentInfo.resolveImageView        = VK_NULL_HANDLE;
        depthStencilAttachmentInfo.storeOp                 = VK_ATTACHMENT_STORE_OP_DONT_CARE;
        depthStencilAttachmentInfo.clearValue.depthStencil = { 1, 0 };
        depthStencilAttachmentInfo.resolveMode             = VK_RESOLVE_MODE_NONE;

        VkRenderingInfo renderingInfo{};
        renderingInfo.sType                = VK_STRUCTURE_TYPE_RENDERING_INFO;
        renderingInfo.pNext                = nullptr;
        renderingInfo.flags                = VK_RENDERING_CONTENTS_SECONDARY_COMMAND_BUFFERS_BIT;
        renderingInfo.renderArea.offset    = { 0, 0 };
        renderingInfo.renderArea.extent    = GState.SwapChainExtent;
        renderingInfo.layerCount           = 1;
        renderingInfo.viewMask             = 0;
        renderingInfo.colorAttachmentCount = 1;
        renderingInfo.pColorAttachments    = &colorAttachmentInfo;
        renderingInfo.pDepthAttachment     = &depthStencilAttachmentInfo;
        renderingInfo.pStencilAttachment   = VK_NULL_HANDLE;

        TransitionImageLayout(GState.ColorImage, GState.SwapChainFormat.format, VK_IMAGE_LAYOUT_UNDEFINED, VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL, 1);
        TransitionImageLayout(GState.SwapChainImages[i], GState.SwapChainFormat.format, VK_IMAGE_LAYOUT_UNDEFINED, VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL, 1);
        TransitionImageLayout(GState.DepthBufferImage, DepthFormat, VK_IMAGE_LAYOUT_UNDEFINED, VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL, 1, true);

        VkCommandBufferInheritanceRenderingInfo dynamicRenderingInheritanceInfo{};
        dynamicRenderingInheritanceInfo.sType                   = VK_STRUCTURE_TYPE_COMMAND_BUFFER_INHERITANCE_RENDERING_INFO;
        dynamicRenderingInheritanceInfo.pNext                   = nullptr;
        dynamicRenderingInheritanceInfo.flags                   = 0;
        dynamicRenderingInheritanceInfo.viewMask                = 0;
        dynamicRenderingInheritanceInfo.colorAttachmentCount    = 1;
        dynamicRenderingInheritanceInfo.pColorAttachmentFormats = &GState.SwapChainFormat.format;
        dynamicRenderingInheritanceInfo.depthAttachmentFormat   = DepthFormat;
        dynamicRenderingInheritanceInfo.stencilAttachmentFormat = VK_FORMAT_UNDEFINED;
        dynamicRenderingInheritanceInfo.rasterizationSamples    = VK_SAMPLE_COUNT_1_BIT;

        VkCommandBufferInheritanceInfo inheritanceInfo{};
        inheritanceInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_INHERITANCE_INFO;
        inheritanceInfo.pNext = &dynamicRenderingInheritanceInfo;
        inheritanceInfo.renderPass = VK_NULL_HANDLE;

        VkCommandBufferBeginInfo CommandBufferBeginInfo{};
        CommandBufferBeginInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
        CommandBufferBeginInfo.pNext = nullptr;
        CommandBufferBeginInfo.pInheritanceInfo = &inheritanceInfo;

        vkCmdBeginRendering(GState.CommandBuffers[i], &renderingInfo);
        {
            vkBeginCommandBuffer(GState.SecondaryCommandBuffers[i], &CommandBufferBeginInfo);

            vkCmdBindPipeline(GState.SecondaryCommandBuffers[i], VK_PIPELINE_BIND_POINT_GRAPHICS, GState.GraphicsPipeline);
            vkCmdBindVertexBuffers(GState.SecondaryCommandBuffers[i], 0, 1, &GState.VertexBufferAllocation.Buffer, Offsets);
            vkCmdBindIndexBuffer(GState.SecondaryCommandBuffers[i], GState.IndexBufferAllocation.Buffer, Offsets[0], VK_INDEX_TYPE_UINT32);
            vkCmdBindDescriptorSets(
              GState.SecondaryCommandBuffers[i], VK_PIPELINE_BIND_POINT_GRAPHICS, GState.PipelineLayout, 0, 1, &GState.DescriptorSets[i], 0, nullptr);
            vkCmdDrawIndexed(GState.SecondaryCommandBuffers[i], Indices.size(), 1, 0, 0, 0);

            vkEndCommandBuffer(GState.SecondaryCommandBuffers[i]);
        }
        vkCmdExecuteCommands(GState.CommandBuffers[i], 1, &GState.SecondaryCommandBuffers[i]);
        vkCmdEndRendering(GState.CommandBuffers[i]);

The application craches in the validation layers, particulary in the vkCmdBeginRendering, looks like it’s trying to dereference a nullptr. Any ideas?

[krOoze]

renderingInfo.pStencilAttachment   = VK_NULL_HANDLE;

Appropriate value for a pointer is NULL or nullptr. It is not a non-dispatchable handle.

---

Check validity of the GState.CommandBuffers[i] dispatch at the point of the call. It must not be NULL or some uninitialized value.

Check validity of GState.ColorImageView, GState.SwapChainImageViews[i], and GState.DepthBufferImageView handles for uninitialized value.

Otherwise layers should not crash without first reporting an error message. In that case report the problem at https://github.com/KhronosGroup/Vulkan-ValidationLayers.

[wookie]

Setting renderingInfo.pStencilAttachment to nullptr didn’t help. Every image/command buffer is valid, as I just added the secondary command buffers to an already working dynamic rendering code.

I guess it has to do something with

Validation layer: Validation Warning: [ VUID_Undefined ] Object 0: handle = 0x1ea4c0b38f0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x79de34d4 | Device Extension VK_KHR_dynamic_rendering support is incomplete, incorrect results are possible.

in the output, didn’t notice it before. I have an RTX 2060 SUPER, maybe something’s is actually missing?

[krOoze]

Right, also don’t forget to update your SDK to the latestest.