Level Editor initialising incorrectly on Release builds (only sometimes!) (C++)

Hi all,

I am having a bit of an odd issue with my OpenGL application.
I’ll first present a rough layout of the application (it’s grown rather big and complex over the past half-year), this will help me describe the issue I’m having.

The application runs the traditional game-loop, switching between game-states. Currently there are two game-states, one for the main menu, and one for the level editor. The main menu is a straight-forward menu navigation system allowing the user to load up the level editor, and thus switch to other game-states.
The GUI for the main menu and GUI items in the level editor are a custom rollout, using vertex array buffers and all that to render it through shaders.

The issue I’m having is the following:
On release builds only, the level editor sometimes will not seem to initialise properly.
What is expected is that the editor loads up a terrain, a skybox and sets up the level editor GUI. The GUI (both for main menu, and level editor) is always visible, however the terrain and skybox (which is just a skydome rendered through a Mesh class) are not always visible/don’t seem to load up correctly. I have a wireframe toggling button, and I don’t see any traces of the terrain or skybox at all, so it seems somehow it’s not loading in the necesarry buffers correctly…

The frustrating part is that, as I said, it only happens on some (release) builds. Sometimes I’ll go in and add a new GUI Element, rebuild, and the terrain and skybox will be visible once again. On Debug builds, I have not experienced this terrain/skybox problem at all (yet, and I run it regularly), so it seems that this is limited to Release builds only.

I’d post code, but the codebase has become quite large, and I doubt people would want to wade through it :slight_smile:

My question is really just a request for guidance, has anyone here experienced similar issues? If so, where should I start looking in order to find the problem?

I’ve tried using debugger tools for windows, and squashed any warnings or errors that gave.
I’ve also tried using windows application verifier, however that always crashes upon the window-creation code (more specifically on the setPixelFormat method), so I can’t let it run any further into the application to find potential heap-issues.

I’m of course willing to post any code if someone is willing to look at specific parts (I’m even prepared to share the SVN repository link in good trust, if necesarry). This problem has been persisting for months now, and it’s about time it gets dealt with :slight_smile:

Thank you very much for any input!

however that always crashes upon the window-creation code (more specifically on the setPixelFormat method), so I can’t let it run any further into the application to find potential heap-issues.

Thats your problem right there. I guess you’re not properly initializing some variables in your code. The debug CRT initializes memory to a certain pattern, but in release your’re usually left with whatever garbage there is upon allocation. This leads to different behavior in debug vs release mode.
Scrutinize all the variables that are getting passed into setPixelFormat for proper initialization.

Hi Skynet and thanks for your reply.

Upon further investigation using application verifier, I have found the following.
When running my release build through the debugger (through visual studio 2012 express edition), with application verifier disabled, it won’t load up correctly.
When running my release build through the debugger, with application verifier enabled, it DOES load up correctly.

For applciation verifier I checked all the ‘Basics’ flags, except Leak (since Leak crashes still upon setPixelFormat, and before this there is really no odd variables being passed in, code posted below).
So it does seem like application verifier also initialises memory along a certain layout, even when running release builds. I do get the impression my issue is not related to setPixelFormat, since the main menu always loads up correctly. Even after extensive testing, it’s never failed to load up correctly there. Are there any useful (preferably free) tools out there that can profile the difference in memory layout between running Debug and Release builds? I’m thinking my incorrect initialisation is probably hidden in some piece of code exclusively related to loading up the Level Editor, but I have searched for days already and no real luck yet.

Here’s my setPixelFormat code:

void GLWindow::setupPixelFormat(void)
    int pixelFormat;

        sizeof(PIXELFORMATDESCRIPTOR),  // size
            1,                          // version
            PFD_SUPPORT_OPENGL |        // OpenGL window
            PFD_DRAW_TO_WINDOW |        // render to window
            PFD_DOUBLEBUFFER,           // support double-buffering
            PFD_TYPE_RGBA,              // color type
            32,                         // prefered color depth
            0, 0, 0, 0, 0, 0,           // color bits (ignored)
            0,                          // no alpha buffer
            0,                          // alpha bits (ignored)
            0,                          // no accumulation buffer
            0, 0, 0, 0,                 // accum bits (ignored)
            32,                         // depth buffer
            0,                          // no stencil buffer
            0,                          // no auxiliary buffers
            PFD_MAIN_PLANE,             // main layer
            0,                          // reserved
            0, 0, 0,                    // no layer, visible, damage masks

    pixelFormat = ChoosePixelFormat(m_hdc, &pfd);
    SetPixelFormat(m_hdc, pixelFormat, &pfd); //<= application verifier crashes right here if the Leak flag is checked under Basics, this happens both in Debug and Release builds

m_hdc is definitely set correctly, as illustrated by this code portion (only posting the relevant part of this function):

LRESULT GLWindow::WndProc(HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
		case WM_CREATE:         // window creation
			m_hdc = GetDC(hWnd);

It would be easier to identify what doesn’t change between the two. Optimisation changes practically everything.

There are two distinct issues here. One is that optimisation (enabled for release builds, disabled for debug builds) changes the memory layout. Typically, stack frames are more compact with optimisation enabled, as optimisation often manages to eliminate some variables and/or share space for variables whose lifetimes don’t overlap. This makes it more likely that a buffer overrun will corrupt something important, rather than e.g. a variable which is no longer live.

The other issue is that debug builds link against debug versions of certain libraries; e.g. MSVCRT, where the debug version initialises heap blocks to a fixed pattern, while the release version leaves it untouched, so it’s more likely to contain zero or near-zero values. If the memory is used as an index or offset, a near-zero value is less likely to result in an address exception.

Unfortunately, I don’t know of any suitable Windows tools other than Application Verifier (but you don’t seem to be having much luck with it). If there are any compilation warnings, address them. Unless you understand the compiler inside-out, don’t assume that any of them are harmless. Modern C compilers are very aggressive at optimisation, and will make full use of the liberties afforded by C’s rules regarding “undefined behaviour” (the rules regarding pointer aliasing often trip up people who think that C is still “portable assembler”). If you have added type casts to eliminate warnings, reconsider the issues behind the warnings (even adding or removing const from a pointer-to-pointer type isn’t safe).

I did indeed have some type casts done to eliminate warnings, I’ll try and go through the more luguber ones and see what I can find.
Will update if/when I find the issue. Thanks for the replies either way!

Hi guys,

I believe I might have finally found the issue, as was probably to be expected it was no issue with OpenGL, but a simple stupid logic error.
I had been experimenting with a deferred rendering scheme in the past, and had set it up so I could toggle between forward rendering and the deferred rendering. For this toggling I had used an enum.
DEFERRED was set to 0, and FORWARD was set to 1. Another boolean was used in other parts of the code to make this distinction as well, however this boolean did not correspondingly automatically set the correct Enum value…

So in short, Deferred rendering code was running even though Forward rendering was expected to be running.

Amazing what a good vacation can do, just got back from my road-trip and bam, immediately find the issue :smiley:
Thanks for the helpful tips!