Proper way to bind D3D11 shared texture handle to OpenGL texture with GL_EXT_memory_object?

I am investigating how to do cross-process interop with OpenGL and Direct3D 11 using the EXT_external_objects , EXT_external_objects_win32 and EXT_win32_keyed_mutex OpenGL extensions. My goal is to share a B8G8R8A8_UNORM texture (an external library expects BGRA and I can not change it. What’s relevant here is the byte depth of 4) with a single mip level allocated and written to offscreen with D3D11 by one application, and render it with OpenGL in another. I am trying to avoid WGL_NV_DX_interop2, because I am hooking into a running game engine to create overlays for it, and need to render with the hooked OpenGL context, ideally without need for texture copy.

First I create my texture in Process A with D3D11, and obtain a shared handle to it, and send it over to process B.

#define WIDTH 100
#define HEIGHT 100
#define BPP 4 // BGRA8 is 4 bytes per pixel

ID3D11Texture2D *texture;

D3D11_TEXTURE2D_DESC texDesc = {
  .Width = WIDTH,
  .Height = HEIGHT,
  .MipLevels = 1,
  .ArraySize = 1,
  .SampleDesc = { .Count = 1, .Quality = 0 }
  .CPUAccessFlags = 0,

device->CreateTexture2D(&texDesc, NULL, &texture);

HANDLE sharedHandle;
texture->CreateSharedHandle(NULL, DXGI_SHARED_RESOURCE_READ, NULL, &sharedHandle);

SendToProcessB(sharedHandle, pid);

In Process B, I first duplicate the handle to get one that’s process-local.

HANDLE localSharedHandle;
HANDLE hProcA = OpenProcess(PROCESS_DUP_HANDLE, false, processAPID);
DuplicateHandle(hProcA, sharedHandle, GetCurrentProcess(), &localSharedHandle, 0, false, DUPLICATE_SAME_ACCESS);

At this point, I have a valid shared handle to the DXGI resource in localSharedHandle. I have a D3D11 implementation of ProcessB that is able to successfully render the shared texture after opening with OpenSharedResource1. My issue here is OpenGL however.

This is what I am currently doing for OpenGL

GLuint sharedTexture, memObj;
glCreateTextures(GL_TEXTURE_2D, 1, &sharedTexture);
glTextureParameteri(sharedTexture, TEXTURE_TILING_EXT, OPTIMAL_TILING_EXT); // D3D11 side is D3D11_TEXTURE_LAYOUT_UNDEFINED

// Setup display parameters to display with ImGui

// Create the memory object handle
glCreateMemoryObjectsEXT(1, &memObj);

// I am not actually sure what the size parameter here is referring to.
// Since the source texture is DX11, there's no way to get the allocation size, 
// I make a guess of W * H * BPP

// According to docs for VkExternalMemoryHandleTypeFlagBitsNV, NtHandle Shared Resources use HANDLE_TYPE_D3D11_IMAGE_EXT
glImportMemoryWin32HandleEXT(memObj, WIDTH * HEIGHT * BPP, GL_HANDLE_TYPE_D3D11_IMAGE_EXT, (void*)localSharedHandle);

Checking for errors along the way seems to indicate the import was successful. However I am not able to bind the texture.

if (glAcquireKeyedMutexWin32EXT(memObj, 0, (UINT)-1) {
  glTextureStorageMem2D(sharedTexture, 1, GL_RGBA8, WIDTH, HEIGHT, memObj, 0); 
  glReleaseKeyedMutexWin32EXT(memObj, 0);

What goes wrong is the call to glTextureStorageMem2D. The shared KeyedMutex is being properly acquired and released. The extension documentation is unclear as to how I’m supposed to properly bind this texture and draw it.

After some more debugging, I managed to get [DebugSeverityHigh] DebugSourceApi: DebugTypeError, id: 1281: GL_INVALID_VALUE error generated. Memory object too small from the Debug context. By dividing my width in half I was able to get some garbled output on the screen.

It turns out the size needed to import the handle was not WIDTH * HEIGHT * BPP, (where BPP = 4 for BGRA in this case), but WIDTH * HEIGHT * BPP * 2. With that, the texture properly binds and can be rendered.