I understand a method of using the stencil buffer to cap a clipped convex shell. The process is to create a stencil by drawing the back facing polygons and then removing the forward facing polygons. Then use the stencil to draw a polygon over the hole left by the clipping plane.
This does not seem to work for more complex objects because forward facing polygons are removed even if they are behind rearward facing polygons.
Is there a way to extend this technique so that it works on concave geometries, or perhaps another way to cap a complex shape?
You haven’t really said what you’re trying to do with the stencil buffer, so the only advise possible is based on a guess at what you’re doing. Your mention of throwing out back faces and breakage of concave geometries makes no sense to me.
HOWEVER, what you have said suggests that you might be doing stencil shadow volume rendering, and you’re dealing with the primary failure mode of what’s termed the “Depth Pass” stencil shadow volume rendering technique.
With Depth Pass, you’re using the stencil buffer to count the number of shadow volumes from the near plane to the framebuffer sample. The main advantage of Depth Pass is that there is no need to render light and dark caps on the shadow volume. The reason is its main disadvantage and a key assumption of Depth Pass, and that is that no part of the near plane may be inside the shadow volume.
To use Depth Pass in cases where the shadow volume intersects the near plane, you end up having to clip the shadow volume and generate nasty caps with all sorts of ugly cases (like the one it sounds like you’re hitting). In general the caps may need to have holes, or holes with holes… etc. Ugly! Even if you aren’t trying to pump out frames at 60Hz.
The oft-touted solution is to flip to Depth Fail (also called Carmack’s reverse) for these cases where the shadow volume intersects the near clip plane. With Depth Fail, instead of counting shadow boundaries from the near plane to the framebuffer sample, you count them from the framebuffer sample to the far clip plane. This turns our near-plane clipping problem into a far-plane clipping problem, which can be more easily dealt with (via depth clamp, infinite far clip, or if you’re a masochist, via generated far-plane caps).
So bottom line, you need to state more about what you’re trying to do with the stencil buffer, and why you’re “throwing away” back faces, so we can give you some advise that’ll be more likely to help.
I am trying to show a cut away view of a model of a person’s teeth and gums.
My approach was to slice it in half using a clipping plane and then cap it off with a stenciled polygon, but as I mentioned the stencil is not being created correctly in certain spots.
To create a stencil of where the clipping plane cuts through the model, I was drawing the backward facing polygons to the stencil buffer, then subtracting the forward facing polygons. This worked for the most part, but in certain spots around the teeth it does not work.
Thank you for the help, I appreciate it. Here is an image of a case that works with my current approach, and a case that fails. The teeth are a little more complicated but hopefully this illustrates what I think the problem is.
The stencil in the second case does not cover the entire hole created by the clipping plane. Then when I try to cap it off with a polygon, the polygon has a hole in it where the stencil is not created correctly.
I think you’ve got the front/back colors swapped on your tooth hole.
In any case, I think I get what you’re trying to do now. Identify the pixels that represent the interior of a closed solid (the tooth) so that you can fill those (i.e. cap the solid).
Seems to me what you want to do increment stencil for back-facing pixels, decrement for front-facing pixels, and anywhere that stencil isn’t zero, you’ve got a “cap” pixel.
This seems to fix your hole problem. In the top picture, your stencil will be 1 for the cap pixels and 0 elsewhere. In the bottom picture. For the “hole” pixels, you have +2-1==1 instead of +1-0==1, but you still end up with 1 (a non-zero value).
This algorithm assumes that the entire solid lies within the near and far clipping planes.