Device lost and multithreading

[luc4s]

Hello,

I have a question about something I don’t understand: I have a thread that submits work to queues (either presenting or command buffers), and other threads can enqueue command buffers to be executed on certain queues (if available, one for graphics, one for presenting and one for transfer).

With a debug build, all runs fine, but in release mode, I get a device lost error when the submitter thread tries to submit a command buffer that copies a buffer to another, and the only way I found to solve that issue was to wait for the last present command to have been submitted (originally it did not wait for the commands to be submitted, but it would use fences to not overwhelm the queues).

Any idea why I get this device lost error? (And the problem does not show up if I don’t use the submitter thread.)

[AndreyOGL_D3D]

Do you have a minimal test case/test project with source code ?
it can help to find problem.

[luc4s]

Unfortunately the project is quite big, here are the main parts though:

Submitter thread:

    void CQueueSubmitter::Enqueue(vk::Queue & queue, CQueueSubmittable_ptr submittable)
    {
        {
            boost::lock_guard<boost::mutex> lock(m_mutex);
            m_pending.push_back(submittable);
            m_queues.push_back(queue);
        }
        m_condition.notify_one();
    }

    void CQueueSubmitter::Run()
    {
        for (;;)
        {
            boost::unique_lock<boost::mutex> lock(m_mutex);
            m_condition.wait(lock);
            while (!m_pending.empty())
            {
                auto & submittable = m_pending.front();
                submittable->SubmitTo(m_queues.front());
                m_pending.pop_front();
                m_queues.pop_front();
                boost::this_thread::interruption_point();
            }
        }
    }

Render loop:

    void CRenderer::Render()
    {
        if (auto infos = m_submitInfos[m_currentFrame])
            infos->Wait(); // Wait until: 1. Submitted 2. Fence signaled

        auto & device = Ctx()->Device();

        uint32_t image_index = 0;
        try
        {
            vk::Fence acquire_fence;

const auto result = device.acquireNextImageKHR(*m_swapchain, UINT64_MAX, *m_imageAvailSemaphore[m_currentFrame], acquire_fence);

            image_index = result.value;
        }
        catch (const vk::OutOfDateKHRError &)
        {
            return;
        }

        m_sceneEncoders[image_index]->Update();

        if (auto submit_info = m_imageSubmitInfos[image_index])
            submit_info->Wait();

        {
            auto submit_fence = m_fences[m_currentFrame];
            submit_fence->Reset();

            auto submit_info = XCore::make_shared<CSubmitInfo>(encoder->CommandBuffer());
            submit_info->SetFence(submit_fence);
            submit_info->SetWaitSemaphores({m_imageAvailSemaphore[m_currentFrame]});
            submit_info->SetSignalSemaphores({m_renderDoneSemaphore[m_currentFrame]});
            submit_info->SetWaitStages({vk::PipelineStageFlagBits::eColorAttachmentOutput});

            auto & queue = Ctx()->GetQueue(eQueueType::kGraphics);
            Ctx()->Submitter()->Enqueue(queue, submit_info);
            m_submitInfos[m_currentFrame] = submit_info;
            m_imageSubmitInfos[image_index] = submit_info;
        }

        {
            auto present_info = std::make_shared<CPresentInfo>(m_swapchain, image_index);
            present_info->SetWaitSemaphores({m_renderDoneSemaphore[m_currentFrame]});
            auto & queue = Ctx()->GetQueue(eQueueType::kPresent);
            Ctx()->Submitter()->Enqueue(queue, present_info);
            present_info->Wait();
        }

        m_currentFrame = (m_currentFrame + 1) % m_framesInFlight;
}

Buffer copy

void CBuffer::CopyFrom(CBuffer_ptr src, size_t size, size_t src_offset, size_t dst_offset)
{
    auto command_pool = New<CCommandPool>(Ctx()->TransferQueueIndex(), true);
    auto command_buffer_ptr = command_pool->CreatePrimary();

    vk::CommandBuffer & command_buffer = *command_buffer_ptr;
    vk::CommandBufferBeginInfo begin_info;
    begin_info.setFlags(vk::CommandBufferUsageFlagBits::eOneTimeSubmit);

    if (size == 0)
        size = Size();

    vk::BufferCopy copy_region;
    copy_region
        .setSrcOffset(src_offset)
        .setDstOffset(dst_offset)
        .setSize(size);

    command_buffer.begin(begin_info);
    command_buffer.copyBuffer(*src, m_buffer, copy_region);
    command_buffer.end();

    auto fence = New<CFence>();
    CSubmitInfo_ptr submit_info = std::make_shared<CSubmitInfo>(command_buffer_ptr);
    submit_info->SetFence(fence);

    auto & queue = Ctx()->GetQueue(eQueueType::kTransfer);
    Ctx()->Submitter()->Enqueue(queue, submit_info);
    submit_info->Wait();
}