Difficult to understand stencil buffer

Hi all:
I read the red book but I still found it is difficult to understand the stencil buffer. How does the stencil buffer draw pictures in a restrict area? And how do the two functions

glStencilFunc(func, ref, mask);
glStencilOp(fail, zfail, zpass);

work? Why the mask value is always some strange value like 0xFFFFFFFF? What is that?

Could anybody explain it in some details please?

Thanks in advance!!!

I’m not really an expert in stencil. Anyway, let’s make a try.

First, this is not the stencil buffer that draws in a restricted area. It’s only used during the GL pipeline process to do some operations on the fragments.

When you enable the stencil buffer, all the fragments go to the chromatic buffer but also go to the stencil one (and of course the depth buffer one).

Regarding the function you specify, fragments will always, never… fill the stencil buffer. It depends on the func you specify but also the mask and the ref. Fragments will never fill in the stencil if the stencil value of that fragment has not its mask bits matching the mask you stipple in StencilFunc.
So if you want your part of the drawing to always fill in the stencil buffer, then you stipple StencilFunc (ALWAYS, 0x1, 0x1).
Now, StencilOp specifies how the incoming fragments will modify the stencil buffer. The threes values you stipple decide what to do when the fragment fails the stencil test, fail to the depth test and succeed to the depth test respectively.
So if for another part of the drawing you want your fragments to modify the stencil buffer only when they succeed to the depth test (taking in consideration previous values in depth test), then you can do something like that:
StencilOp (KEEP,KEEP,NEW_VALUE*) where NEW_VALUE must be a value different from what you had, ie incrementing…

Hope that could help.

Thanks jide. It makes more sense to me. But still confused about what is the meaning of the reference and mask value like 0x1? The data type is GLint and GLuint respectively and what’s the sense of the 0x1? How should I decide the reference and mask value then?
Also, when the stencil buffer is enabled, the fragments are drawed in color buffer and stencil buffer at the same time. What happens if I disable the stencil buffer?

The reference value is what is put into the stencil buffer for a fragment that succeed the test and when you stippled GL_REPLACE.

For more understanding about masks, I’ll guess a good book about programming could help you.

They are not drawned at the same time. Depth occurs first, then stencil then the color buffer.

Since stencil buffer is a collection of bitplanes (usually 8), each plane can hold one bit (0 or 1). If the mask is 0x1, then the stencil test will be performed only on the first bitplane. If the mask is ~0x0 (or 0xFFFFFFFF), then all planes will be tested.

Usually, you render to stencil buffer only in the first pass to make a restricted rendering area, then render to color buffer in the second pass by referencing the stencil values. For example, draw to color buffer only if the fragment is passed the stencil test.

I found the description about half way down the nehe tutorial #26 useful in understanding stencil buffers.