Stencil buffer question

Hi,
I’ve been looking for good stencil buffer tutorials online but can’t get a handle on the whole thing.
I’m trying to create a circular matte for a viewport.
Basically I’m trying to achieve something like glScissor but with a circle rather than a rectangle.
Could anyone provide a simple example?
Thanks!

You should call glClearStencil(0) somewhere in your code so that the stencil buffer gets zero when glClear is called.


//Clear
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT);

//Allow writing to all bits in the stencil buffer
glStencilMask(0xFFFFFFFF);
//Always pass the stencil test
glStencilFunc(GL_ALWAYS);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
glEnable(GL_STENCIL_TEST);

//Now render whatever shape you want. Example : A circle.
//The stencil will be 1 where the circle is drawn
RenderMyCircle();

//Now, we want only the framebuffer to be updated if stencil value is 1
glStencilMask(0xFFFFFFFF);
glStencilFunc(GL_EQUAL);
//Do not change stencil
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glEnable(GL_STENCIL_TEST);

//Render your other stuff
RenderMyStuff();

Hey,
Thanks for your help. This is not working for me. Here is the code I’m using:


glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT);

//Allow writing to all bits in the stencil buffer
glStencilMask(0xFFFFFFFF);
//Always pass the stencil test
glStencilFunc (GL_ALWAYS, 0x1, 0x1); //glStencilFunc(GL_ALWAYS);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
glEnable(GL_STENCIL_TEST);

//Now render whatever shape you want. Example : A circle.
//The stencil will be 1 where the circle is drawn
glColor4f(1.0,1.0,1.0,1.0);
gl::drawSolidCircle(Vec2f(0.0,0.0),50.0);

//Now, we want only the framebuffer to be updated if stencil value is 1
glStencilMask(0xFFFFFFFF);
glStencilFunc (GL_EQUAL, 0x0, 0x1); //glStencilFunc(GL_EQUAL);
//Do not change stencil
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glEnable(GL_STENCIL_TEST);

//Render your other stuff
glColor4f(1.0,0.0,0.0,1.0);
gl::drawSolidRect(Rectf(0.0,0.0,25.0,25.0));

The glStencilFunc’s you provided wouldn’t compile for me, so I replaced them… Is that correct?

Here is what the above code looks like rendered:
http://tinypic.com/r/2jakq6g/7

What am I doing wrong?

Thanks.

You can set the mask and reference to 0xFFFFFFFF
glStencilFunc(GL_ALWAYS, 0xFFFFFFFF, 0xFFFFFFFF);
glStencilFunc(GL_EQUAL, 0xFFFFFFFF, 0xFFFFFFFF);

Hey,
Thanks for your help.
That seems to make no difference. I’m getting the same results as before. Any ideas?
Thanks!

You probably want to disable writing to the color buffer when drawing your stencil shape:

glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

And then re-enable drawing to the color buffer to draw the rest:

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

If you want to write inside the stencil shape, then your second glStencilFunc should probably be:

glStencilFunc (GL_EQUAL, 0x1, 0x1);

And if you are going to be forming a complex stencil shape with multiple draw calls, then it may be better to use

glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

because otherwise you may have values >1 in the stencil buffer if you draw to the same pixel of the stencil buffer multiple times.

Oh, and have you remembered to request a stencil buffer at context creation time?

A couple of articles/tutorials that might be helpful, although quite advanced:
http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=26
http://www.slideshare.net/Mark_Kilgard (improving shadows + reflections via the stencil buffer)

Hey, thanks.
Got it!