Validation Error: VkCommandBuffer is already in use

Vulkan
1

What I am trying to accomplish is an offscreen render in VK_FORMAT_R32G32B32A32_SFLOAT and convert to VK_FORMAT_R8G8B8_UNORM for presenting to the swapchain window surface: An offscreen to swapchain hand off.

The validation error I am getting is:

ERROR: [774851941][VUID-vkQueueSubmit-pCommandBuffers-00071] : Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] Object 0: handle = 0x2a8c400, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] VkCommandBuffer 0x311ee30 is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT

To be clear, I am not trying to use VkCommandBuffer(s) in parallel or simultaneous use.

The error tells me I am doing something wrong with my buildCommandBuffers() function: ( @krOoze )
My buildCommandBuffers() function:

void VulkanSystem::buildCommandBuffers()
{
	VkCommandBufferBeginInfo cmdBufInfo = {};
	cmdBufInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;

	for (int32_t i = 0; i < drawCmdBuffers.size(); ++i)
	{
		VK_CHECK_RESULT(vkBeginCommandBuffer(drawCmdBuffers[i], &cmdBufInfo));

		/** TODO: Check if it would be more efficient to draw once outside of this command buffer loop */
		{
			/*
				OFFSCREEN FRAMEBUFFER
			*/

			std::array<VkClearValue, 3> clearValues;
			clearValues[0].color        = {{0.0f, 0.0f, 0.0f, 0.0f}};
			clearValues[1].color        = {{0.0f, 0.0f, 0.0f, 0.0f}};
			clearValues[2].depthStencil = {0.0f, 0};

			VkRenderPassBeginInfo renderPassBeginInfo    = {};
			renderPassBeginInfo.sType                    = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO;
			renderPassBeginInfo.renderPass               = offscreen.renderpass;
			renderPassBeginInfo.framebuffer              = offscreen.framebuffer;
			renderPassBeginInfo.renderArea.offset.x      = 0;
			renderPassBeginInfo.renderArea.offset.y      = 0;
			renderPassBeginInfo.renderArea.extent.width  = _width;
			renderPassBeginInfo.renderArea.extent.height = _height;
			renderPassBeginInfo.clearValueCount          = 3;
			renderPassBeginInfo.pClearValues             = clearValues.data();

			vkCmdBeginRenderPass(drawCmdBuffers[i], &renderPassBeginInfo, VK_SUBPASS_CONTENTS_INLINE);
			
			VkViewport viewport = {};
			viewport.width = (float)_width;
			viewport.height = (float)_height;
			viewport.minDepth = 0.0f;
			viewport.maxDepth = 1.0f;
			vkCmdSetViewport(drawCmdBuffers[i], 0, 1, &viewport);

			VkRect2D scissor = {};
			scissor.extent.width = _width;
			scissor.extent.height = _height;
			scissor.offset.x = 0;
			scissor.offset.y = 0;
			vkCmdSetScissor(drawCmdBuffers[i], 0, 1, &scissor);

			// 3D Mesh Object: DescriptorSet and Pipeline
			vkCmdBindDescriptorSets
			(
				drawCmdBuffers[i],
				VK_PIPELINE_BIND_POINT_GRAPHICS,
				pipelineLayouts.primary,
				0,
				1,
				&descriptorSets.primary,
				0,
				nullptr
			);

			// VkPipeline usePipeline = _enable_sampleRateShading ? pipelines.MSAASampleShading : pipelines.MSAA;
			VkPipeline usePipeline = pipelines.primary;
			vkCmdBindPipeline(drawCmdBuffers[i], VK_PIPELINE_BIND_POINT_GRAPHICS, usePipeline);
		
			// 3D Mesh Object: Bind Vertex and Index Buffers
			_mesh.bind(drawCmdBuffers[i]);
			_mesh.draw(drawCmdBuffers[i]);

			// End of render pass: wrap up
			vkCmdEndRenderPass(drawCmdBuffers[i]);
		}


		{
			/*
				SWAPCHAIN FRAMEBUFFER: PRESENT TO DISPLAY
			*/

			std::vector<VkClearValue> clearValues;
			for (uint32_t i = 0; i < _displayDevice->_bufferCount; i++)
			{
				VkClearValue clearVal{};
				clearVal.color = {{0.0f, 0.0f, 0.0f, 0.0f}};
				clearValues.push_back(clearVal);
			}

			VkRenderPassBeginInfo renderPassBeginInfo    = {};
			renderPassBeginInfo.sType                    = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO;
			renderPassBeginInfo.framebuffer              = framebuffers[i];
			renderPassBeginInfo.renderPass               = renderPass;
			renderPassBeginInfo.renderArea.offset.x      = 0;
			renderPassBeginInfo.renderArea.offset.y      = 0;
			renderPassBeginInfo.renderArea.extent.width  = _width;
			renderPassBeginInfo.renderArea.extent.height = _height;
			renderPassBeginInfo.clearValueCount          = clearValues.size();
			renderPassBeginInfo.pClearValues             = clearValues.data();

			vkCmdBeginRenderPass(drawCmdBuffers[i], &renderPassBeginInfo, VK_SUBPASS_CONTENTS_INLINE);

			VkViewport viewport = {};
			viewport.width = (float)_width;
			viewport.height = (float)_height;
			viewport.minDepth = 0.0f;
			viewport.maxDepth = 1.0f;
			vkCmdSetViewport(drawCmdBuffers[i], 0, 1, &viewport);

			VkRect2D scissor = {};
			scissor.extent.width = _width;
			scissor.extent.height = _height;
			scissor.offset.x = 0;
			scissor.offset.y = 0;
			vkCmdSetScissor(drawCmdBuffers[i], 0, 1, &scissor);

			vkCmdBindDescriptorSets
			(
				drawCmdBuffers[i],
				VK_PIPELINE_BIND_POINT_GRAPHICS,
				pipelineLayouts.display,
				0,
				1,
				&descriptorSets.display,
				0,
				NULL
			);

			// Scene
			vkCmdBindPipeline(drawCmdBuffers[i], VK_PIPELINE_BIND_POINT_GRAPHICS, pipelines.display);
			vkCmdDraw(drawCmdBuffers[i], 3, 1, 0, 0);

			vkCmdEndRenderPass(drawCmdBuffers[i]);
		}

		VK_CHECK_RESULT(vkEndCommandBuffer(drawCmdBuffers[i]));
	}

I have two renderpasses and pipelines, one is simply rendering the scene offscreen, and the second is doing an image sampler from the first renderpass.

This is my draw step:

void VulkanSystem::draw()
{
	if (!prepared) { return; }

	// Acquire the next swapchain image (swapBuffers part 1)
	VK_CHECK_RESULT(_displayDevice->acquireNextImage(semaphores.presentComplete, &currentBuffer));

    // Draw
	submitInfo.commandBufferCount = 1;
	submitInfo.pCommandBuffers = &drawCmdBuffers[currentBuffer];
	VK_CHECK_RESULT(vkQueueSubmit(queue, 1, &submitInfo, VK_NULL_HANDLE));

	// Present the frame (swapBuffers part 2)
	_displayDevice->queuePresent(queue, currentBuffer, semaphores.renderComplete);
}

I appreciate any help, I am still fairly new to Vulkan (since November) and I have been stuck on this problem for the last month and a half. (Most of my career has been OpenGL)

Please let me know if more parts of my code is needed to help answer this question.

Thank you in advance!

Respectfully,
John

2

No, the error tells you that you are submiting the command buffer without retiring its previous submission.

Since your submit does not even reference any fence, then it is quite likely the layers are correct.

3

Thank you for the correction and clarification.

Since your submit does not even reference any fence, then it is quite likely the layers are correct.

I know this is entirely a seperate question: Should I use fences? The example I’m following is HDR in Vulkan Examples by the Khronos Group, and it does not use fences (it is at least not apparent to a beginner like me). Before integrating the HDR example this was my previous code in my render engine utilizing fences:

void VulkanSystem::draw() 
{
	if (!prepared) { return; }

	// Acquire the next swapchain image	
	VK_CHECK_RESULT(_displayDevice->acquireNextImage(semaphores.presentComplete, &currentBuffer));

	// Wait for the fence to ensure the previous frame is finished
	vkWaitForFences(device->logical, 1, &renderFences[currentBuffer], VK_TRUE, UINT64_MAX);
	vkResetFences(device->logical, 1, &renderFences[currentBuffer]);

    // Prepare and submit the render frame
	submitInfo.commandBufferCount = 1;
	submitInfo.pCommandBuffers = &drawCmdBuffers[currentBuffer];

	// Submit the command buffer and specify the fence for the current frame
	VK_CHECK_RESULT(vkQueueSubmit(queue, 1, &submitInfo, renderFences[currentBuffer]));

	// Present the frame
	_displayDevice->queuePresent(queue, currentBuffer, semaphores.renderComplete);
}

Back to the previous question: Let’s just say that my buildCommandBuffers() step is correct, how do I retire a previous command buffer?

My guess is something kind of like this for my draw step:

vkQueueSubmit(queue, 1, &submitInfo, VK_NULL_HANDLE);
vkQueueWaitIdle(queue);

// Reset the command buffer for reuse
vkResetCommandBuffer(commandBuffer, 0);
4 
vkQueueWaitIdle

The use of this function is almost always indicative of a mistake somewhere.

I’m not familiar with the example code you’re using as a template, but it may not recycle command buffers at all. Or it might be using counting semaphores to track execution.

In any case, what matters is that you must do something to check to see if the GPU is finished with a batch of work containing the command buffer in question before it can be used. Fences are a typical approach (especially since they’re needed by other things too), but not the only one.

5

@Alfonse_Reinheart , thank you for the reply.

I’m not familiar with the example code you’re using as a template, but it may not recycle command buffers at all. Or it might be using counting semaphores to track execution.

The example code was from the GitHub KhronosGroup Vulkan-Examples. It is under the samples/api/hdr directory.

I am now trying a different API approach but same example by @Sascha_Willems’ HDR example from his GitHub repository. This might help since my render engine is more loosely based off of @Sascha_Willems’ render engine. I believe I missed something with descriptor setup or subpass dependencies somewhere that is not allowing me to use the same command buffer to utilize multple renderpasses in my buildCommandBuffers() function call. I start and end renderpasses on the same command buffer.

Before attempting to do this offscreen rendering example, (only using 1 renderpass), the code works without validation layer errors because I am only using 1 renderpass per command buffer.

6

I switched gears to follow @Sascha_Willems HDR example instead of the Khronos Group HDR example. Since my render engine is loosely based off of his API, it worked. I think there was an integration problem since the Khronos Group Vulkan Examples has a different API.

I am sorry that I was not able to solve the particular issue in this thread. My educated guess is that I was trying to force color attachments to match the number of swapchain buffers and command buffers as well as all the descriptor pool size, layouts, and sets, and it lead to some confusion and a mismatch in command buffers executing in the queue.