Using the stencil buffer for the first time

OpenGL OpenGL: Advanced Coding
1

I want to use the stencil buffer to mask out my background in a deferred renderer, so the full-screen directional light pass only draws to pixels that have had something drawn to them, and for other things. So far it is not working in a predictable way.

The stencil buffer is cleared like this:

glStencilMask(0xFF);
glClearNamedFramebufferfi(fbo, GL_DEPTH_STENCIL, 0, 1.0f, 0);
glStencilMask(0x1);

I prepare to draw objects like this:

glEnable(GL_STENCIL_TEST);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0x1);

Now I get ready to draw the full-screen pass for directional lights:

glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFunc(GL_EQUAL, 0, 0xFF);// Background pixels get drawn to
//glStencilFunc(GL_EQUAL, 1, 0xFF);// Nothing gets drawn
glStencilMask(0);

If I use glStencilFunc(GL_EQUAL, 0, 0xFF) then the full-screen pass draws to the background pixels (where nothing has been drawn).

image
image1282×752 46.5 KB

This indicates that the FBO is correctly set up with an active stencil buffer attached.

However, calling glStencilFunc(GL_EQUAL, 1, 0xFF) does not draw lighting on my foreground pixels (where objects have been drawn). I just get a black screen instead.

image
image1282×752 3.69 KB

What am I doing wrong?

2

Even stranger, this will cause the background to be written to:

glStencilFunc(GL_NOTEQUAL, 1, 0xFF);

But this will cause nothing to be written to:

glStencilFunc(GL_EQUAL, 1, 0xFF);
3

Using glReadPixels I was able to verify the stencil buffer contains a large number of 1 values, zero values, and nothing else. So it’s just the second drawing step that is not working.

auto buffer = CreateBuffer(dest->size.x * dest->size.y);
		buffer->Clear();
		glReadPixels(0, 0, dest->size.x, dest->size.y, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, buffer->Data());
		for (int n = 0; n < buffer->GetSize(); ++n)
		{
			auto b = buffer->PeekByte(n);
			if (b != 0) Print(b);
		}
4

I thought for sure this was an Intel driver bug, but the same exact behavior occurs on an AMD 6600.

5

Pure guess:

Is the code which draws the foreground writing to the depth buffer so that anything subsequently drawn to the foreground fails the depth test?

6

No, depth testing is disabled when the full-screen pass is drawn.

7

This was not the cause, but it got me thinking that maybe the stencil is working, and something else was wrong…I forgot to disable stencil testing after the fullscreen pass, and there’s another pass to present that to the screen, which was failing since it had no stencil buffer.

It’s working now. Thank you.

image
image1282×752 66.4 KB