I cannot delete this texture because a command buffer uses a "Device Memory" that cannot be freed

It’s normal if a vulkan report is very big because it’s the hardest backend, and I spend many hours on this post.

// Step 1: Delete textures.
{
    SAFE_DELETE(_internalFonts._fontAtlasTexture); // <-- The texture I want to delete.
    SAFE_DELETE(_iconsTexture);
}
// Step 2: Delete the command buffer.
SAFE_DELETE(_copyCmd); // <-- This just deletes the class CopyCommandBuffer but the real vulkan command buffer inside it is already ended before.

Goal: I will always repeat the following words, when I say “Texture that cannot be deleted”, I’m talking about Vulkan memory leak, which means that the texture (class Texture) is not completely deleted.

For example, if the call below fails, then it means that the texture cannot be deleted, because there are some vulkan objects that cannot be freed / deleted / destroyed.

// In the function Texture::Vulkan::destroy()
Texture *texture = this;
vkFreeMemory(device, texture->_memory, nullptr);

The error on that line above:
[vulkan] Debug report from ObjectType: VK_DEBUG_REPORT_OBJECT_TYPE_DEVICE_EXT
Validation Error: [ VUID-vkFreeMemory-memory-00677 ] Object 0: handle = 0x27e7d8679a0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x485c8ea2 | Cannot call vkFreeMemory on VkDeviceMemory 0x908683000000001d[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution (Link: [Vulkan® 1.2.182 - A Specification (with all registered extensions)] I cannot include link because I’m a beginner).

The thing I don’t understand:

  • Cannot call vkFreeMemory on VkDeviceMemory 0x908683000000001d[] that is currently in use by a command buffer.
  • The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution.

My own interpretation, cannot free a “Device Memory” because it’s currently in use by a command buffer. But according to the codes below, I try to END / FLUSHING it like this _engine->_copyCmd->end(); and does it means that the command buffer doesn’t use the “Device Memory” anymore, I’m not sure?

The command buffer:

//****************************************************************************************************
// CopyCommandBuffer
// Referenced from "examples/imgui/main.cpp" > initResources in "Vulkan C++ examples and demos" https://github.com/SaschaWillems/Vulkan
//****************************************************************************************************

VkCommandBuffer VulkanDevice::createCommandBuffer(VkCommandBufferLevel level, VkCommandPool pool, bool begin)
{
    VkCommandBufferAllocateInfo cmdBufAllocateInfo = vks::initializers::commandBufferAllocateInfo(pool, level, 1);
    VkCommandBuffer cmdBuffer;
    VK_CHECK_RESULT(vkAllocateCommandBuffers(logicalDevice, &cmdBufAllocateInfo, &cmdBuffer));
    // If requested, also start recording for the new command buffer
    if (begin)
    {
        VkCommandBufferBeginInfo cmdBufInfo = vks::initializers::commandBufferBeginInfo();
        VK_CHECK_RESULT(vkBeginCommandBuffer(cmdBuffer, &cmdBufInfo));
    }
    return cmdBuffer;
}
void VulkanDevice::flushCommandBuffer(VkCommandBuffer commandBuffer, VkQueue queue, VkCommandPool pool, bool free)
{
    if (commandBuffer == VK_NULL_HANDLE)
    {
        return;
    }

    VK_CHECK_RESULT(vkEndCommandBuffer(commandBuffer));

    VkSubmitInfo submitInfo = vks::initializers::submitInfo();
    submitInfo.commandBufferCount = 1;
    submitInfo.pCommandBuffers = &commandBuffer;
    // Create fence to ensure that the command buffer has finished executing
    VkFenceCreateInfo fenceInfo = vks::initializers::fenceCreateInfo(VK_FLAGS_NONE);
    VkFence fence;
    VK_CHECK_RESULT(vkCreateFence(logicalDevice, &fenceInfo, nullptr, &fence));
    // Submit to the queue
    VK_CHECK_RESULT(vkQueueSubmit(queue, 1, &submitInfo, fence));
    // Wait for the fence to signal that command buffer has finished executing
    VK_CHECK_RESULT(vkWaitForFences(logicalDevice, 1, &fence, VK_TRUE, DEFAULT_FENCE_TIMEOUT));
    vkDestroyFence(logicalDevice, fence, nullptr);
    if (free)
    {
        vkFreeCommandBuffers(logicalDevice, pool, 1, &commandBuffer);
    }
}
void VulkanDevice::flushCommandBuffer(VkCommandBuffer commandBuffer, VkQueue queue, bool free)
{
    return flushCommandBuffer(commandBuffer, queue, commandPool, free);
}

emc::CopyCommandBuffer::CopyCommandBuffer(Engine *engine) {
    _engine = engine;
    _isBegan = false;
    _vk_copyCmd = nullptr;
}
emc::CopyCommandBuffer::~CopyCommandBuffer() {
    EMC_VERIFY(!_isBegan);
    EMC_VERIFY(_vk_copyCmd == nullptr);
}
void emc::CopyCommandBuffer::begin() {
    EMC_VERIFY(!_isBegan);
    _isBegan = true;
    _vk_copyCmd = _engine->_vk._vulkanDevice->createCommandBuffer(VK_COMMAND_BUFFER_LEVEL_PRIMARY, true);
}
void emc::CopyCommandBuffer::end() {
    EMC_VERIFY(_isBegan);
    _isBegan = false;

    // Free
    if (_vk_copyCmd) { _engine->_vk._vulkanDevice->flushCommandBuffer(_vk_copyCmd, _engine->_vk._copyQueue, true); _vk_copyCmd = nullptr; }
}

How I create the texture (I cannot delete the texture _fontAtlasTexture):

_engine->_copyCmd->begin(); // <-- Begin the Command Buffer.
    if (!_fontAtlasTexture) {
        emc::TextureCreateInfo CI = {};
        CI.type = emc::TextureCreateInfoType_Blank;
        CI.extent = { extent.x, extent.y, 1 };
        _fontAtlasTexture = new Texture(_engine, _engine->_copyCmd, CI);
    }
    else {
        _fontAtlasTexture->resize(extent);
    }

    // Map & Write & Unmap
    emc::TextureMappedResource mappedResource = _fontAtlasTexture->map();
        memcpy(mappedResource._rgba_data, fontData, uploadSize);
    _fontAtlasTexture->unmap();
_engine->_copyCmd->end(); // <-- End the Command Buffer.

Note: There are many textures but only that texture (_fontAtlasTexture) which cannot be deleted. And it’s hard for me to detect where is the problem because the codes are almost with the same method, for example, read the code below:

How I create the icons texture (I can delete the texture _iconsTexture without memory leak):

TextureCreateInfo CI = {};
CI.type = TextureCreateInfoType_FromFile;
CI.fileName = binPath(_projectName) + "../data/icons/icons.png";
_iconsTexture_size = { 12, 12 };

_copyCmd->begin(); // <-- Begin the Command Buffer.
    _iconsTexture = new Texture(this, _copyCmd, CI);
_copyCmd->end(); // <-- End the Command Buffer.

Not sure what is there to understand. The error message cannot be more clear. You must not free memory that is still in use. But let’s see.

(Link: [Vulkan® 1.2.182 - A Specification (with all registered extensions)]

Always update your SDK to the latest. It increases coverage and reduces false positives and other bugs.

The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution.

Your image\buffer is tied to a command buffer that was submitted but have not yet finished execution. You as an application are obligated to ensure the command buffer have finished execution before calling vkFree*, otherwise it is undefined behavior.

I try to END / FLUSHING it like this _engine->_copyCmd->end();

Do or do not. There is not try. :stuck_out_tongue:

If you do not understand your own abstractions, then I am even less likely to understand them. Either this end() function does block on command buffer before it finishes, or it does not.

You have to identify all uses of the resource in your code and make sure all of the uses have finished.

For debugging purposes you can try vkDeviceWaitIdle. If it does not work, then it points to some larger issue. Then you can try vkQueueWaitIdle to identify which queue is the problem. Then find all submits on that queue, and determine if all of them are terminated by the time the vkFree* is called. You could use VK_LAYER_LUNARG_api_dump which will deabstract the code and gives a stream of Vulkan commands as they are actually executed, which could be used to overview what the lifetime of that memory object looks like.

1 Like

In addition to what has already been said, your use of the Vulkan API exhibits several practices of a dubious nature.

Any object wrapping a command buffer (which is already not a good idea) should not have a member function for submitting it to a queue. This API encourages multiple submit calls into the same queue, which is a terrible idea. It’s so bad that the Vulkan specification takes time out to tell you how bad of an idea it is:

Submission can be a high overhead operation, and applications should attempt to batch work together into as few calls to vkQueueSubmit or vkQueueSubmit2 as possible.

Putting submission in the command buffer gives the impression that it’s OK to submit CBs one after the other.

Additionally, textures should not have a 1:1 equivalence relationship with memory allocations. Multiple textures should be stored in the same allocation. Or more to the point, you should make a few allocations, then subdivide them to provide storage for individual textuers/buffers.

As such, “deleting a texture” should almost never also delete the underlying memory providing storage for that texture. So ideally, you should not have encountered this problem at all.

1 Like

I updated to the current latest Vulkan 1.3.204.1, and I compare the Validation Errors with the old Vulkan but they are identical, with a little error descriptions (text) modification for the latest Vulkan.

Thank you for the replies about vkFree* and vkQueue*, they will help me when I will become more advanced because I’m still a beginner. Some advanced codes I showed in the first post are not written by me, I only wrote the codes with namespace emc::.

If I analyze the text in the image at bottom, then it concludes:

  • Cannot call vkFree*/vkDestroy* on VkX that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to X must have completed execution.

The thing I don’t understand:

  • How to complete execution of submitted commands that refer to X ? Can someone explain that? Because if I don’t understand it then I have no idea where to go, where to debug, etc. According to my experience, giving links to examples on how to do such thing is a good idea, because as I said before, I use codes written by professional developers.
#define VK_SAFE_DESTROY(x, p) { if (p) { vkDestroy##x(device, (p), nullptr); (p)=nullptr; } }

The overall problem is that you shouldn’t want to do this. Not the way you’re trying to do it.

Basically, what you’re trying to do is this. Some code in your system says “I’m done with this Vulkan object”. So at that moment your system decides to delete it.

That last part is the problem: you should not delete Vulkan objects whenever you want. You stop using them, then at some future time, you delete them when Vulkan is done with them.

If you absolutely have to delete these things in a relatively quick manner (and again, see my point above about how bad this code is. Also FYI: Windows implementations of Vulkan limit the number of distinct allocations to 4096 specifically to prevent pathological allocation behaviors like what you’re doing), you should store such objects somewhere in a “to-be-deleted” pile. When you store them, each bundle of objects should be associated with a particular fence representing the last queue-submit call that uses those objects.

At the start of every frame, you can check to see if the fence associated with a bundle of objects has been signaled. If it has, this means that all work submitted with that fence (or in submit commands to that queue before the fence) has completed. At which point, it is safe to delete those objects.

Of course, you also need to make sure that any such objects stop being used by other objects. If they’re in descriptor sets, they have to be removed from those sets. And modifying descriptor sets can have synchronization issues too.

But overall, this represents sloppy handling of Vulkan resources, and should only be used in situations where you absolutely have no other choice.

1 Like

If you absolutely have to delete these things in a relatively quick manner

Thank you. I delete objects because my program/application has a memory leak explosion, the total number of not freed memory blocks doesn’t stop increasing, it never stops, and I think that if there are objects that cannot be deleted then such memory leak can happen.

Thank you, I will think about those replies, maybe I will come back after a few days, Vulkan is too difficult.

You need to keep track of all the submissions that refer to it.

The execution of a submission is complete when you vkWait* on the VkFence that was provided to the submission.

For debuging purposes or outside hotspots one can also use vkQueueWaitIdle or vkDeviceWaitIdle, which will ensure all submissions to a queue (resp. all queues of the device) have finished execution.

1 Like

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.