wglDXRegisterObjectNV OutOfMemory in 12000x6000 OpenXR mode (GPU memory looks fine)

This is probably too obscure, but I’m having an annoying problem with a VR set when using OpenXR in max-super-sampling mode. I can switch into VR (allocate frame buffers, etc.) one or two times before I get a ERROR_OPEN_FAILED error from this extension, glGetError reports OutOfMemory which I assume isn’t a translation from the Win32 error code to a GL error code and it really means the context is out of some kind of memory.

It’s not out of GPU memory because there’s plenty in the graph in Window’s Task Manager program and it doesn’t look like it’s leaking.

Is there a limit on a GLRC (WGL context) memory usage of some kind? Can it get fragmented? The old textures are being deleted. The memory usage pattern should be stable. It doesn’t happen with the default super sampling level that is a about half this (I rounded down to 1000s before.)

Is this just to be expected on a mid-tier Nvidia (3060) card? Thanks if anyone has any insight or ideas I can try out.

The extension’s documentation says this means it couldn’t open the Direct3D resource for sharing, but I suspect that it’s too much of a coincidence that glGetError says OutOfMemory.

A couple of thoughts for you.

First, since you’re on an NVIDIA GPU using OpenGL, I’d be sure to plug in a KHR_debug message callback (Details Here). Why I say that: the NVIDIA driver may be trying to provide you more detailed info on the cause of the GL error. As a relevant for-instance, GL_OUT_OF_MEMORY is sometimes emitted by the NVIDIA driver when you run out of some limited memory resource. For instance, hitting the annoying MSWindows-only 4096 max allocations limit gets you one of those (under Vulkan, this is VkPhysicalDeviceLimits::maxMemoryAllocationCount). GL_OUT_OF_MEMORY is the basic GL error thrown, but the extended error message sent to the GL debug message callback further qualifies it as: Exceeded buffer residency limits - buffer count

As for wglDXRegisterObjectNV() errors with VR-related usages. I’ve actually hit errors thrown by this call when running with a different VR runtime (sounds like you’re using WMR at least in one of your other recent threads; this OTOH was on Varjo). But basically this call was erroring out trying to create/register a GL texture share handle for a DX texture and failing because the DX texture resided on a different GPU than the one the GL context was created on. That’s likely not your issue. But could be it’s erroring out due to some error on the DX side trying to create such a high resolution texture or renderbuffer, or merely trying to register such a high resolution texture/renderbuffer with GL (possibly exceeding some implementation-specific width/height limit). For more details on this API, see WGL_NV_DX_interop and WGL_NV_DX_interop2.

Not sure if any of this is useful, but hope it gives you some ideas to pursue.

1 Like

I am trying to follow-up but am trying to finish a VR demo too :smiley:

I had that debug callback set up but not with logging, and its breakpoint got deleted.

Unknown internal debug message. The NVIDIA OpenGL driver has encountered an out of memory error. This application might behave inconsistently and fail.

I think it’s using the GCC philosophy that if you run out of physical RAM you just kill the process instead of using virtual memory. The app itself has 300MB paged, but some apps like Firefox are using more. It’s not GPU memory. System memory is at 28.6/33.2GB “committed” but this shouldn’t be a problem. I guess it thinks it needs raw memory for something special.

EDITED: FWIW it keeps spamming this message.

Could be. You might not be running out of paged but possibly some other resource, such as CPU pinned memory (aka non-paged, paged-locked, etc.), contiguous GPU device memory, or some kernel/driver-internal resource. More on that here by Detlef Roettger @ NVIDIA:

I wrote a lengthy reply (a not very helpful one) but by the end I got an idea to try the “Use latest preview OpenXR runtime” checkbox. It seems like the problem is in the WMR runtime because twice I was able to transition in and out like 4 or 5 times before running into the same pattern, whereas before it was typically 1 or 2 times.

Also I thought I was seeing it yesterday when I change my code to change the resolution without transitioning… it would sometimes go black (except for the runtime FPS display… that really sucks because it’s pinned to the VR display and doesn’t scale with the resolution) but not crash… suggesting something else failed on the runtime’s side… just depending on when memory ran out I guess. But I worked on some things yesterday that might have looked like this but weren’t since today I can cycle resolution without crashing… and only transitioning out of VR crashes (i.e. to the Cliff House and back.)

I’m still seeing sometimes black screens and sometimes crashing. After the full resolution screen goes black I can still switch to the smaller screen and get a picture. A single RGBA8 buffer at this scale would be about 300MB for 2 eyes worth. There’s probably a few of these, and a depth buffer and MRT buffer of the same size… but it really doesn’t seem like a problem in GPU memory because it never gets very large. It seems like spending time in the Cliff House and then switching back can guarantee it to happen.

This is probably enough speculation! It is quite annoying.