Help with Stencil Test

OpenGL OpenGL: Basic Coding
1

Hi guys,

I’m following the stencil tutorials from learnopengl.com and I tried something and I can’t explain why is not working. I might not understood very good the stencil test.
So I have a floor and 2 containers and I want to draw a border around containers, like in the tutorial. The code will be:

//Before the rendering loop:
glEnable(GL_DEPTH_TEST);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

In the rendering loop:
//Draw the floor but don’t write to stencil buffer
glStencilMask(0x00);
DrawFloor();

//Draw the 2 containers
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0xFF);
DrawTwoContainers()

//Scale containers for the borders
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00);
glDisable(GL_DEPTH_TEST);
DrawTheTwoScaledContainers()

glStencilMask(0xFF);
glEnable(GL_DEPTH_TEST);

Well everything is working fine, but at the code where I scale the containers, if I change the GL_NOTEQUAL, with GL_LESS, it draws only the normal containers, but not the borders and not even the floor. Why is this happening ? The stencil values where the borders should be drawn are 0, so they are not equal and they are less than 1.

Hope you can help me with an explanation and thanks in advance!

2

Short answer: You probably got confused which value stand on which side during the comparison: https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glStencilFunc.xhtml

Long answer:

Have a look at this pseudo code for the stencil test:


maskedBufferValue = bufferValue & stencilFunctionMask;
maskedStencilRefValue = stencilRefValue & stencilFunctionMask;

if (NOT StencilCompFunc(maskedBufferValue, maskedStencilRefValue))
    discardFragment()

...

UpdateStencilBuffer(hasStencilTestPassed, hasDepthTestPassed, stencilMask);

So first we use the mask (3. value of glStencilFunc) with a bitwise AND operation on the value currently stored in the stencil buffer and on the specified reference value (2. value of glStencilFunc). If you set the mask to 0x00, both result values are always 0. If you set it to 0xFF it results in a plain assignment of the original values. So in your case it is not that important since you are only using 0xFF.

So lets walk through the original example:

First you draw the floor. StencilFunc at this point is: glStencilFunc(GL_NOTEQUAL, 1, 0xFF), since it is the last call to this function in the previous loop iteration. Stencil mask is 0x00 which disables writing to the stencil buffer. So assuming that you cleared the stencil buffer correctly (see pitfall at the end), the following happens here:
Since the stencil function mask (don’t get confused with the stencil write mask) is 0xFF we take the unmodified values for our comparison. It is the first draw after clearing. The buffer values are 0 everywhere. So you are always comparing 0!=1 which results in true. Only if this function would have yielded a false, the fragment would have been discarded. So as a result, everything passes and is drawn. The stencil buffer is not updated, since the stencil mask is 0x00.

Now we get to the containers. We use glStencilFunc(GL_ALWAYS, 1, 0xFF) and glStencilMask(0xFF). The stencil comparison will always yield true and the values are not modified. So we get straight to the buffer update. Since the stencil test passed there are now two possible results: The depth test fails and you will do nothing since you specified GL_KEEP as second param of glStencilOp or the depth passes and the 3. action specified by glStencilOp is used for the update which is GL_REPLACE. This states that the reference Value of glStencilFunc replaces all the bits specified by stencil mask in the current buffer. In this case, all bits are replaced (0xFF). The result is, that every fragment which belongs to a container and passes the depth test has now a stencil value of 1, while the rest is zero.

Last we get to the scaled containers glStencilFunc(GL_NOTEQUAL, 1, 0xFF), glStencilMask(0x00). So a stencil passes if bufferValue != 1. This is everywhere except for the location where the original container was rendered and the fragment passed the depth test. This results in an outline since the containers are scaled up and you can only draw at positions that are not part of the original container. The stencil buffer is not updated.

Now what happens, if we set GL_LESS instead of GL_NOTEQUAL?
Floor: glStencilFunc(GL_LESS, 1, 0xFF) — All values in the buffer are 0 and the refValue is 1. Values after the mask are the same. 1 < 0 always fails. This is why your floor is never drawn.
Containers: Same as before.
Scaled Containers: glStencilFunc(GL_LESS, 1, 0xFF). Two possible cases for the stencil test: 0 < 0 (fail) or 1<0(fail). Scaled container fragments can never pass the stencil test!

Hope that helps understanding why you get the described result.

PITFALL: When I was implementing the stencil stuff I searched half of the day for an error in my code which was caused by a missing glStencilMask(0xFF) before clearing the stencil buffer. The stencil mask seem to affect the clear command too. So if it is set to 0x00 a call to glClear(GL_STENCIL_BUFFER_BIT) has no effect. Keep that in mind, it might save you the same trouble I had :wink: — I asked myself, if this is a bug, but if you think about it makes sense. This can be used to clear only specific bits from the stencil buffer allowing you to do really crazy things.

3

Wow, thank you very much for your explanation.
I didn’t understood very well the AND operation. I thought it was first with the stencil value and then with the mask and just compare which one is bigger. :smiley: But now it is more clear.
Thank you very much again. :smiley:

4

[QUOTE=alx119;1293648]Wow, thank you very much for your explanation.
I didn’t understood very well the AND operation. I thought it was first with the stencil value and then with the mask and just compare which one is bigger. :smiley: But now it is more clear.
Thank you very much again. :D[/QUOTE]

Glad I could help :wink: