Concave Polys & Stencil Buffer?


I am trying to use the stencil buffer to render simple concave
polygons (when detected). I found the algorithm i’m using online -
but I think I don’t have it right yet! Either my concave poly
disappears or it gets drawn as before (i.e. with artifcats).

This my code snippet…

// Draw poly as set of fan tris - ie 123-
// 134-145 etc.
// draw into the stencil buffer
glStencilFunc(GL_ALWAYS, 0, 1);

int num_tris = num_children - 2;
for( int i=0; i<num_tris; i++ )

// then draw poly as normal but stencil
// should limit drawing into the concave
// parts. This time turn on drawing

glStencilFunc(GL_EQUAL, 1, 1);



I am sure the stencilop & func are wrong but
I’m not that familiar with the stencil
buffer to know what im doing wrong…?

Any help appreciated.


Concave polygons are not supported by OpenGL, so if DrawFace() is not doing exactly the same as DrawTri() you’re screwed.
Replace DrawFace() with the DrawTri() loop.

Anyway this is only working for flat shading.
The real solution is to write a concave polygon triangulation algorithm.
One very easy algorithm checks the sign of the dotproduct of two edges at one vertex and if it has the right orientation (counterclockwise or clockwise) draw that triangle and take the vertex out of the polygon list.
Goto the next vertex.
Stop if only two vertices are left.

[This message has been edited by Relic (edited 07-23-2003).]

Thanks - i’ll think about that- any triangulation routine or decomposition is going to be more effort than a pixel solution. The algorithm that i’m trying to implement is mentioned in the red book so I’m sure its a valid approach - however i think my selection of some of the stencil parameters is making it fail. I was hoping for someone to spot the flaw & set me right - without having to write/restructure my code.

Thanks for the different approach tip - if I cant get the right call set using the stencil buffer I might be doing it that way after all!:slight_smile:


I think the stencil buffer calls are ok in your code.
I don’t know what DrawFace() does. If it draws a fullscreen rectangle, ok. If you draw the same as for setting the stencil, ok. Both should work, IF you have a stencil buffer.

Check what you get with glGetIntegerv(GL_STENCIL_BITS, &i);

Ok, read chapter 14 of the readbook.
Pass 1:
The initial drawing into the stencil buffer is done with a single(!) triangle fan.
Make sure you have face culling off.
Disable depth writes if you have depth test enabled, otherwise pass 2 might fail.
Pass 2:
The redraw should be done with some polygon fully containing the convex hull.

[This message has been edited by Relic (edited 07-23-2003).]


Im pretty much doing (i think) what the book says (if I dont do the stencil stuff & only draw the fan - it looks right! and I can see the true concave poly shape nicely). My problem is knowing what to set the stencilfunc & stencilop for each polygon “pass”. When I experimented with values it was obviously doing “something” - sometimes nothing would render, sometimes the poly - still with concave artifacts would be drawn. This leads me to believe that the right combination of params would lead to the desired result. I just havnt hit on the right values yet i’m afriad… :frowning: