WGL - Using Stencil Functins Mess Up Depth Testing

Hi Folks:

I'm at the half way point in [these](https://learnopengl.com/#!Advanced-OpenGL/Stencil-testing) great tutorials.  I've been refining a library of my own for rendering Blender models for the last several weeks.  

I’ll use this model for a crash test dummy as I work my way through the second half of the tutorials.

Thanks to everybody in this forum who’ve helped me get this far.

I’ve been running into a problem with the “Stencil Test” tutorial, which the link points to.

Depth testings seems to go nuts when I try to do anything with a stencil.

Here are the models I’m rendering in a splash screen:

The program’s name “Net Results” should be drawn in the background, with the tennis court rotating in front of it. The giant golf ball is a temporary part of the tennis court model, to help me see if the lighting effects are working.

This looks right when running the splash screen code.

I’ve added that purple box and placed it on the playing surface for the stencil tutorial.

The tutorial’s exercise is to use the stencil test to draw a yellow outline around the purple box.

The OpenGL related include section looks like this:

#include <GL/glew.h>
#include <GL/wglew.h>

// GLM Mathemtics
#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>
#include <glm/gtc/type_ptr.hpp>

I’m using assimp, so there’s also a reference to that header somewhere.

I’ve tried this several times, and watched it fail. Before the stencil code is added I test everything but. The purple box is being drawn, the yellow box that will become the outline is being drawn around it. Everything looks good.

Let’s do stencils.

Here are the stencil related code fragments I found from the sample program, and how I fit them into my code:

In the setup function:

glStencilFunc(GL_NOTEQUAL, 1, 0xFF);

glDepthFunc(GL_LESS) seems redundant, but it was in the sample.

In the loop:

GL_STENCIL_BUFFER_BIT was added to the glClear() argument.

“Net”, “Results” and the tennis court are drawn.

Before the purple box is drawn:

glStencilFunc(GL_ALWAYS, 1, 0xFF);

After the box is drawn, preparing to draw the outline:

glStencilFunc(GL_NOTEQUAL, 1, 0xFF);

The depth test is disabled to draw the border so it will be visible even if it’s behind something.

I then select the outline’s shader, it’s fragment shader just returns yellow. Then scale the box’s model transform matrix into the outline’s model matrix.

Set some uniforms in the outline’s shader and draw it.

After drawing:


And the depth testing seems to be toast.

Here’s another shot:

“Net” and “Results” are drawn first, if no depth testing was in effect I believe these would be drawn behind everything else. Next the court is drawn, then the purple box and finally, with GL_DEPTH_TEST disabled, the yellow outline. Yet the outline is the element that’s written over by almost everything.

The first suspect was the enabling and disabling of GL_DEPTH_TEST. I removed all the stencil stuff and kept the depth test stuff, images displayed fine.

Commenting out the stencil statements, then uncommenting them one by one shows the depth testing goes bad with the glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE) statement.

I know this function uses the results of a depth test fail. Do I need a different argument for it?



What’s your value for glClearStencil? Also, in OpenGL (and contrary to what you might expect) clears are affected by the current buffer writemasks; what this means is that if glStencilMask is 0 then the stencil buffer will not be cleared. Either of thse could be messing up your expected result.

you could solve this easily without stencil test:

/* dont write into depth buffer */

/* draw background text here */

/* allow writing into depth buffer again */

/* draw everything else */

if you want to learn how stencil test works, and thus doing it wit stencil test:

  1. clear all buffers, including stencil buffer
  1. draw your scene (here: the tennis playing field etc…). fill every pixel with a pre-defined value

GLint value =   0x00 00 00 01;
GLuint mask = 0xFF FF FF FF;
glStencilFunc(GL_ALWAYS, value, mask);

GLenum sfail = GL_KEEP;
GLenum dpfail = GL_KEEP;
GLenum dppass = GL_REPLACE;
glStencilOp(sfail, dpfail, dppass);

/* draw your scene here */

NOTE: you have to keep in mind that your stencil buffer has limited precision, thats why the value will be clamped to [0; 2^N - 1], where N = stencil bits
for example: 8 bits -->max value = 255 = 0x000000FF

  1. draw the background: stencil test still enabled, but the function changes, that is stencil test should pass if the value is NOT EQUAL to (value & mask)
glStencilFunc(GL_NOTEQUAL, value, mask);

/* and the stencil buffer should NOT be updated, because you want to use it as a "foreground mask" */
sfail = GL_KEEP;
dpfail = GL_KEEP;
dppass = GL_KEEP;
glStencilOp(sfail, dpfail, dppass);

/* draw here your background stuff, whatever it might be */

didnt test that, but that should work (… hopefully! ;))

glClearStencil() is initially 0
step 1 fill the stencil buffer with 0s everywhere
step 2 sets the value to 0x00000001 & 0xFFFFFFFF = 0x00000001 where your foreground is drawn
step 3 only checks if buffer value != value (which is 0x00000001 & 0xFFFFFFFF = 0x00000001), and discards the fragment if so

Thanks mhagain:


I’m now setting it to 0xFF with the idea that 0 is the “No write” stencil value for a fragment. I have no clue if that’s a bad idea.

I’m far from understanding these subtleties.

But the fog is beginning to lift, as you’ll see in my response to John Conner, which might not be finished before I crash after working on this all night.

I appreciate this advice.


Thanks John:

I’m writing this after working on this program all night. If this post becomes incoherent, that might be a contributing factor.

I now suspect what I thought was a depth test issue was a case of items being rendered setting stuff in the stencil buffer.

Thanks for showing me a different approach to stencils than that taken in the tutorials.

Joey DeVries has done an amazing job with this set of OpenGL tutorials, but perhaps this particular topic could use a little work. I think the subject of stencils is just confusing to newbies by it’s nature.

I couldn’t get your stuff to work quite right either, but it did provide the foundation I used to get close to what I’m looking for.

Is there a way to upload these images to this server? I’ve had to link to images on my web site.

I could certainly call it a victory and move on, but I’d like to understand stencils better before looking at more interesting tutorials.

So my assignment is to present that box with that outline from any angle. It’s not working quite right:

Things drawn in front of the box should have the outline burned through them, but the box it surrounds should be displayed with the same depth testing as any object in the scene.

So the image shouldn’t have a section of the ball, or the net, drawn in yellow, except where they are in front of the border.

Here’s what I’m doing. Please let me know where I’m messing up.

I've pulled all stencil related code from the startup.  It's all in the loop now.  

I've decided to initialize the stencil buffer to 0xFF, with 0x00 the "Don't draw here" value.  

1) So I want to clear the stencil buffer with 0xFF:

before calling glClear()

I diverge from your example in that I draw everything up to the box without invoking anything about stencils.

  1. Before drawing the box:

// The stencil buffer is cleared with 0xFF.

// We'll draw the box over any value in the stencil buffer and leave 0xEE.
glStencilFunc(GL_ALWAYS, 0xEE, 0xFF);

// If a fragment passes all the tests, and is drawn, 
// replace the stencil's mask value for that fragment with 
// the glStencilFunc()'s reference value.  That's set up to 
// not draw fragments with that value.
  1. Before drawing the outline:

// We'll draw the outline over anything with a stencil value that's less than the box.
glStencilFunc(GL_LESS, 0xEE, 0xFF);

  1. After the outline is drawn:
  1. I haven’t done this yet, but I want to be able to render other items, perhaps some making use of the stencil, after the box and it’s outline.

  2. At the end of the loop:


So that’s where I’m at now, thanks again for helping me get this far.

I was hoping this:

// We'll draw the outline over anything with a stencil value that's less than the box.
glStencilFunc(GL_LESS, 0xEE, 0xFF);

would prevent the yellow that’s splashed over anything rendered in front of the box and border.

Also, I’m not understanding where I use glStencilMask(). As I understand it statements like:

glStencilFunc(GL_EQUAL, 1, 0xFF)

dictate the value written to the stencil buffer. What part does the glStencilMask() function pay in that.

I could go on, but I really need to crash. I’ll look at this post after I wake up to see if it makes any sense.

Thanks again John, and mhagain, for your assistance.


just to reiterate how te stencil test works:

there is a stencil buffer with certain values in it. if you’ve cleared it. it will contain the value GL_STENCIL_CLEAR_VALUE. (use glGetIntegerv() to get it, use glClearStencil() to set it)

the stencil test must be enabled to have any effect:

there is a value called GL_STENCIL_VALUE_MASK. (use glGetIntegerv() to get it, use glStencilMask() or glStencilFunc() to set it)

once you draw something, you generate a lot of fragments, each has a certain “fragmentvalue”. (use glStencilFunc() to set it as well as mask)

then it works like this:
fragmentvalue = fragmentvalue & mask;
buffervalue = buffervalue & mask;
thats why 0xFF is a good mask to start with --> it effectively skips these 2 operations

bool stenciltest = fragmentvalue FUNC buffervalue;
FUNC is a operator, you set it via the glStencilFunc(…)

2nd part: what to do if the stencil test fails / passes ?
thats what you can define with glStencilOp(sfail, dpfail, dppass)

if (stenciltest)
    if (depthtest)
    { stencilbuffervalue = ... what dppass indicates ... }
    { stencilbuffervalue = ... what dpfail indicates ... }
    stencilbuffervalue = ... what sfail indicates ...

GL_KEEP says keep the stencilbuffers value
GL_REPLACE says replace the stencilbuffers value with “fragmentvalue”

here’s the example i’ve posted before:

Thanks John:

I decided to walk through the stencil code in your link with Visual Studio 2017’s debugger.

VS 2017 is confused.

There are several instances of enums like this:

enum Queries {
    /* none */

Followed by GLuint array declarations like this:

GLuint queries                     [MAX_Queries];

Which produce error messages like this:

error C2466: cannot allocate an array of constant size 0

I’m still recovering from the all night grind, so I’ll probably figure it out. Any hint’s would be appreciated.

  Thanks Again
/* objects sharable between contexts */
GLuint buffers			[MAX_Buffers + 1];
GLuint queries			[MAX_Queries + 1];
GLuint renderbuffers	[MAX_Renderbuffers + 1];
GLuint samplers			[MAX_Samplers + 1];
GLuint textures			[MAX_Textures + 1];
GLuint programs			[MAX_Programs + 1];

/* container objects */
GLuint framebuffers			[MAX_Framebuffers + 1];
GLuint transformfeedbacks	[MAX_TransformFeedbacks + 1];
GLuint vertexarrays			[MAX_VertexArrays + 1];
/* NOTE: +1 necessary to avoid compile errors if MAX_... = 0 */

yepp, got that same error with VS 2017, i’ve coded that with codeblocks + minGW, there it works just fine. just add 1 to each array, doesnt matter at the end of the day how big these arrays, as long as they can hold at least MAX_… elenemts. OpenGL will silently ignore glGen*(MAX_…, …) and glDelete(MAX_…, …) if MAX_… = 0.

the convenient thing about this coding style is that you dont have to create new variables for new GL objects, just add a enum above and prepare it in the “void Initialize()” function (by the way: picked that style up in “openGL programming guide 8th edition”)

Thanks again John:

I think I now know the basics, what you describe in response #6 and what you demonstrate in the code sample you linked, but I still have a question.

And I still don’t now how to solve the stencil problem I’m currently having with my little project.

The question:

  What does the glStencilMask() function do?

  You specify the stencil mask as the third argument in your two calls to glStencilFunc().  

  You never call glStencilMask().  

  I understand, glStencilMask() sets a stencil mask somewhere.  When is the mask set with glStencilMask() used?  What is it's purpose if you're going to specify the stencil mask as the third argument in the call to glStencilFunc() anyway?  


I think I understand glStencilMask() now.

glStencilFunc()'s mask argument is for testing. glStencilMask() masks what is written with glStencilOp(). Calling glStencilMask() isn’t necessary in your example because the mask defaults to 0xFF.


Which leads to this question:

Why doesn’t glStencilOp() just have a fourth argument for the mask, which would make glStencilMask() unnecessary?


I’m going to work on my little stencil project for a bit. I have some ideas to try.

Basically, I want to figure out how to have fine control of the stencil test as I draw multiple objects, some masked some not, in the scene.

As always, thank you for your efforts.