Trouble reading stencil buffer


I’m having difficulty reading the stencil buffer. I’ve written code using glut as well as a Windows specific app just to be sure the glut library I was using wasn’t the problem. In both cases, my problematic code can be distilled down into the following:

   GLenum error = 0;
    printf("Saving image...");
    GLint iViewport[4];

    glGetIntegerv(GL_VIEWPORT, iViewport);
    // Get stencil buffer
    size_t stencilBuffSize = iViewport[2] * iViewport[3];
    GLubyte *stencilBits = new GLubyte[stencilBuffSize];
    GLubyte *end = stencilBits + stencilBuffSize;

    // Read bits from color buffer
    glPixelStorei(GL_PACK_ALIGNMENT, 1);
    glPixelStorei(GL_PACK_ROW_LENGTH, 0);
    glPixelStorei(GL_PACK_SKIP_ROWS, 0);
    glPixelStorei(GL_PACK_SKIP_PIXELS, 0);

    // Clear the window with current clearing color
    error = glGetError();

    error = glGetError();
    glReadPixels(0, 0, iViewport[2], iViewport[3], GL_STENCIL_INDEX,
                 GL_UNSIGNED_BYTE, stencilBits);
    error = glGetError();
    bool pass = true; 
    for (size_t idx = 0; idx < stencilBuffSize; idx++)
        if (stencilBits[idx] != 0)
            pass = false;

    if (pass)
        ::MessageBox(0, "Reading Stencil Buffer test passed!",
                        "glrect", MB_OK);
        ::MessageBox(0, "Reading Stencil Buffer test failed!",
                        "glrect", MB_OK | MB_ICONEXCLAMATION);


My main development system is a HP xw6200 workstation with an ATI FireGL v3100. I’m running XP64 with the latest 8.163.1 video drivers from ATI. I’m compiling this code using Visual Studio .NET 2003.

On this system, the code above indicates reading the stencil buffer fails. The buffer should be all zeros as a result of the call to glCear().

When debugging, viewing the contents of stencilBits shows that some of the buffer contains zeros, but other parts of the buffer contains non-zero data.

If I run the same code, on the same machine, against the Microsoft Software OpenGL driver, the code passes. If I run the same code on a Windows XP box with an NVidia Quadro video card, it passes. If I run similar code under GLUT on a Linux box with the Mesa OpenGL library, it also passes.

My actual use of the stencil buffer is, obviously, far beyond what I’m doing here, but this code is simple enough to show the problem.

My conclusion, at this point, is that ATI’s FireGL drivers are providing incorrect data for the stencil buffer. I’m prepared to approach ATI with this unless someone can point out any coding errors I’ve made.

A complete copy of my test app can be provided upon request. It doesn’t do much else beside initialize a Window and draw several lines on the screen.


Jeff McWilliams