I have a custom game engine that I wrote that our game runs on with both iOS and Android using OpenGL ES 2.0.
Lately, I had gotten a lot of complaints of crashing on Android. After about a full week of trying to trace down why we had memory pressure crashes from graphics memory (seen via Android profiler), I managed to isolate the issue to an extent, but it does not seem like a single trackable flaw. It almost seems like it’s in their drivers.
It occurs only in the background thread rendering, but many of the issues when I try to isolate them, seem to be random. For instance, I may background render a preview for something which contains components A, B, C, D, and E, and if I remove a single one or group of those components, it will not leak, but then if all of them are together, it still leaks. I should also note that checking openGL errors yields nothing. The foreground render and the background render do not share any resources and use a totally different instance of our OpenGL manager. Does anyone have any tips on how to fix this?
Ok, I “Might” have a solution. It at least stabilized the app for me. I appears that prior to calling usleep on the background thread, you need to call glFinish() to avoid the leak. This seems very random, but it is what is working for me.
First thing is to isolate the cause. Then, consider what options you have to fix it.
I have an idea what might be going on. But first, when you say “memory pressure crashes from graphics memory (seen via Android profiler)”, you’re talking about the number by Graphics: here, right?:
Yes, it would climb through the crash. Every time my preview generation code would work, it would deterministically leak on specific designs without any noticeable cause. Our OpenGL Manager class would show that all resources were being correctly released. We have a thread dependent OpenGL manager so there were two different context and they would both independently use different textures and buffers. On handoff, the background thread would read the images out to client memory and then thread safe hand it off to the front process that would bring it into its context
As I said in my second post, the solution, which does not seem like it should be necessary, is that I need to do a glFinish before running any sleep command on the background thread. Talking to other programmers, it seems like multithreaded Android graphics programming is very unstable, so perhaps this is just poorly written drivers.
Ok. So you’re doing a GL “readback to app CPU memory” on one thread and a GL “upload from app CPU memory” on another thread?
If you’re uploading to textures from the CPU continuously like this, this can trigger ghosting of the texture resources in the driver. Basically, you end up with multiple “physical” textures consuming memory being managed behind one GL texture handle. The driver is doing this texture ghosting to try and avoid blocking GL queuing via your CPU thread(s). The con is that it implicitly consumes more GPU (driver graphics) memory behind-the-scenes, allocating more and more memory for these ghosted resources.
If you don’t avoid or at least carefully limit how many of these ghosted resources are created, you can easily blow out your GPU/graphics memory.
Also, GL readbacks can trigger blocks in submission, as the GPU has to catch up to be able to service your request. This too is a big deal if perf matters to you. It can easily cut your frame rate in 1/2 (or worse).
Well, if that workaround is good enough for you, then you can go with that. That’s almost certainly going to reduce your performance.
Depending on the driver, it could be. I’ve definitely worked on some mobile GPU drivers that “leave a lot to be desired” (diplomatically). That said, I’ve also worked on some really solid, awesome ones too!
However, it could instead be that the developer doesn’t really understand how a mobile GPU works. There are certain things that you “can” do via GL-ES but that (given knowledge of how a mobile GPU and a mobile GPU driver works) you should “never” do. As they trigger problems exactly like you’re seeing (excessive growth in GPU/driver memory, very poor performance, 1/2 the frame rate, visual artifacts like full-screen flashing, mid-frame flushes, MSAA downsampling artifacts, blocking in the driver due to resource contention, etcetc.)
Unlike desktop GL/GPUs, with mobile GL/GPUs you “really” need to understand what’s going on under-the-hood. The GPU+system doesn’t have the performance or spare memory to virtualize over costly usage mistakes.
So this Ghosting that you speak of is very interesting. I would completely understand if it kept extra texture memory around to avoid having to reallocate. This bug however was much more extensive than that. I have my engine tuned so that internally, it will normally consume ~800MB of texture memory and then being purging, but this bug would cause it to grow unbound to over 3GB before it crashed due to memory pressure. At some point, the GPU seemed to be losing reference of what it had allocated. I have a lot of OpenGL experience, but I am not familiar with how it works under the hood. Regardless, this seems like it would be out of spec.
Overall, this process is generating a preview of a user created object. This is a decently long process which includes opening, and sometimes downloading multiple assets with many texture maps, as well user color customizations. Because of this, the cost of a GLFinish at the end is very low in comparison, so all I really need it to do is simply work without leaking.
