glStencilFunc and size of the stencil buffer

The function glStencilFunc describes what OpenGL should do with the content of the stencil buffer. It has the signature:

void glStencilFunc(GLenum func,
  	GLint ref,
  	GLuint mask);



Specifies the test function. Eight symbolic constants are valid: GL_NEVER , GL_LESS , GL_LEQUAL , GL_GREATER , GL_GEQUAL , GL_EQUAL , GL_NOTEQUAL , and GL_ALWAYS . The initial value is GL_ALWAYS .


Specifies the reference value for the stencil test. ref is clamped to the range [0,2n−1]

, where n

is the number of bitplanes in the stencil buffer. The initial value is 0.


Specifies a mask that is ANDed with both the reference value and the stored stencil value when the test is done. The initial value is all 1’s.

I just saw the possible options for the func GL_NEVER,GL_LESS,GL_LEQUAL,GL_GREATER,GL_GEQUAL,GL_EQUAL,GL_NOTEQUAL, andGL_ALWAYS. The initial value isGL_ALWAYS` . And they all are boolean(sort of) operations resulting in either 1 or 0 .

If the stencil test decides to whether write or not write a certain fragment, unlike depth buffer not storing a value in range [0,1] , why is the size of the stencil buffer 8 bits and not 1 bit?

question 2:
Most of the time in learnopenGL(learnopengl. com/Advanced-OpenGL/Stencil-testing) the value of glStencilFunc's ref parameter is 1. notice that this is of type GLint so it can be negative, if so why?

question 3:
with reference to the questionn 1 and question 2 , if the operation is only boolean like and the value can be negative or any other 8bit feasible value(that 8 bits can accommodate), then what can other values be and why would we use them?

question 4:
with reference to question 3, and the function glStencilOp(GLenum sfail, GLenum dpfail, GLenum dppass) if the function contained, glStenilFunc(GL_EQUAL, 1, 0xFF) and glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE), then, we’d store a 0000 0001 in the stencil buffer? Wasting 7 bits? or is it converted to: 1111 1111? if this conversion is done, then what would be the conversion for other numbers like 0101 0010(82)(which is in between 1 and 256

Look at the options for glStencilOp. In general, the stencil buffer is used to store a winding number. If you want to fill based upon even-odd winding, you only need one bit. Other winding rules (non-zero, positive, negative) require more bits.

There doesn’t seem to be much point, as the reference page states

The stencil buffer is used for insideness-testing, either in 2D (e.g. non-convex, possibly self-intersecting polygons) or 3D (constructive solid geometry aka CSG; light/shadow volumes are a simple example of this).

There are no conversions. The stencil buffer stores an 8-bit value (typically; nothing prevents an implementation from supporting stencil buffers with more than 8 bits). The stencil test function (glStencilFunc) specifies a test which passes for some values and fails for others.

Even if you’re only using GL_REPLACE as the stencil operation, having multiple bits allows you to generate 8 different 1-bit masks then use a combination of them for the test. But the main reason for having multiple bits is the use of GL_INCR and GL_DECR; operations which use these will typically fail if the count overflows, so you need enough bits to store the worst-case winding number.

one thing I recently came across, does the ref parameter(2nd parameter) in glStencilFunc(GL_ALWAYS, 1,0xFF) not matter?

If glStencilOp uses GL_REPLACE for any of the three cases, then that case will store ref in the stencil buffer. Neither ref nor mask affect the stencil test if the function is GL_ALWAYS or GL_NEVER.

but, from docs .gl the mask value is:

Specifies a mask that is ANDed with both the reference value and the stored stencil value when the test is done

so, it should affect when we GL_ALWAYS or GL_NEVER if it is specified something other than 0xFF, right?

If you use GL_ALWAYS, the stencil test always passes regardless of the values of mask and ref.

If you use GL_NEVER, the stencil test never passes regardless of the values of mask and ref.

For any other option, the test passes if

(value & ref) op (mask & ref)

where op is one of <, <=, >, >=, ==, or !=.

I just forgot to ask. It does always pass regardless of the value of mask and ref when using GL_ALWAYS or GL_NEVER , but I meant if we do glStencilFunc(GL_ALWAYS, 0xAA,0xAF) and we have glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE)` even though the test passes, the value in the stencil buffer is modified, right? I mean it’s not ignoring all the ANDing and replacing operations in those two cases(GL_ALWAYS and GL_NEVER), right?

Yes. The stencil buffer will be set to 0xAA (modified by the glStencilMask value; the mask parameter to glStencilFunc only affects the test, not the update).

If the stencil test fails, neither the colour buffer(s) nor the depth buffer are modified; the stencil buffer will still be modified if the first argument to glStencilOp isn’t GL_KEEP.

Also, this was a typo:

It should be:

(value & mask) op (ref & mask)
1 Like