Looks like I have to bump this. Those “odd artifacts” don’t appear to be going away.
I don’t really know what’s causing them but I do know that I want them gone. Is there a way I can disable self-shadowing using shadow volumes? For what it’s worth, it appears that they appear mainly in concave areas.
For what it’s worth, it appears that they appear mainly in concave areas.
Is that one of the weaknesses of this shadow volume technique?
I thought it might be, but I really can’t find it. And thinking of how shadow volumes work (extrusions of the silhouette), I don’t see why these artifacts should be there.
We’re visual people. 
Let’s see some pictures of the artifacts.
Like I mentioned before, there are limitations and constraints that shadow volumes imposes, and there are failure cases in that particular shadow volume technique that you need to get a solid handle on. If your artifact isn’t a coding bug you’ve introduced, my guess is that one of those constraints is being violated or failure case is being exercised.
First, you mentioned your objects often have cracks. That’s a big no-no. With shadow volumes, you have to be able to ensure that every pixel (or every sample, when rendering with MSAA) will be updated by every front facing and back-facing shadow volume face that should affect it – no exceptions! To do that, you need to ensure that your extruded shadow volume silhouettes are perfect. And to ensure that you typically need a welded mesh. That is, a mesh where for each location there is exactly one vertex in the mesh representing it, and there are no cracks/T-junctions (all vertices on edges split that edge)! The silhouette alg on the NeHe page presumes this, though doesn’t state it. See “What happens is that you go through every face, and if it is visible, …”
Further the Z-pass technique on that page has some failure cases. First, you can’t let any shadow volume intersect the near plane inside the frustum. If you do, shadow artifacts results. Think through it and you’ll see why. The near plane clips away shadow volume pixels the eye can see, messing up the in-shadow/out-of-shadow count in the stencil buffer. You can generate near caps to try and avoid this, or you can use Z-fail stencil shadows in this case. This flips your near-clipping problem into a far clipping problem, and there are better options for dealing with that problem. While you can generate far caps, you also have the option of using “depth clamping” to have the GPU generate them automagically, or you can use an infinite far clip projection matrix which eliminates the possibility of far plane clipping.
Some silhouette determination techniques only work for convex casters. Look through how NeHe’s doing that carefully and make sure it works for concave casters. Reading this may be useful:
And thinking of how shadow volumes work (extrusions of the silhouette), I don’t see why these artifacts should be there.
When you determine your silhouette edges, draw the scene from the light’s perspective and draw the silhouette edges in red. That may give you some insight. Failing that, draw your shadow volume tris in red wireframe, and rotate around the object to see if it looks good. I suspect you’ll see cracks or missing faces.
Also, try punting your existing casters and use some very simple convex surfaces to verify that your alg works in the base case: cube, etc. Then try a really simple shape with a concavity.
So, with a cube it was fine, but the problems begin to arise with an icosphere. All faces not facing the light (mostly, all faces beyond the silhouette) are being shadowed. I don’t want that. How to prevent?
I’ll show you what I mean: 
And with a complex model:
I don’t understand. With direct lighting on volumetric surfaces, all surfaces not facing the light “are” in shadow.
I think maybe one problem here is that you aren’t doing lighting, or aren’t doing it per-pixel, or aren’t doing it correctly. Near the edges of your shadows, the surface normal is orthogonal to light rays, so dot(N,L) == 0, which means the diffuse lighting there is 0. So basically, at least as far as diffuse is concerned, the component of your lighting that gets shadowed should already have faded out to 0 at the point where your shadows start (i.e. where object turns back-facing in light space). So your light back-faces won’t be so blatently obvious at at all. This will make your image look tons better, especially if you do fragment lighting or have enough verts so that vertex lighting doesn’t look so bad. Also make sure your normals are normals for the surface being approximated, not normals for individual triangles.
Also, keep in mind you only shadow diffuse and specular, not ambient or emission. I’d just use ambient and diffuse for now to keep life simple.
This should get your object lighting and self-shadows looking much better and not so faceted and slightly confusing as they do now.
I am just using ambient and diffuse and I’m only shadowing from the diffuse lighting.
The sphere rather explains the problem. You see how all of the sphere’s faces beyond the silhouette are put into shadow? That shouldn’t happen: the sphere’s faces are lit properly if I don’t shadow them. A face shouldn’t shadow itself.
I don’t understand this statement. For a convex object like this, all the faces completely beyond the silhouette (from the perspective of the light) are back-facing and thus not directly lit. They happen to be in shadow, but it’s totally irrelevant that they are because the diffuse lighting contribution there is 0 – so there’s no diffuse light to shadow!
I guess I don’t completely understand that sphere picture, or exactly what you’re saying is wrong with it. It does look like it’s chopped off in a very strange way. But without good lighting or a light position or anything, it’s hard to tell what we’re looking at.
The reason it looks chopped off is because the shadow volume is affecting the back faces of the sphere when it shouldn’t be. That’s what I meant when I say that those faces shouldn’t be in shadow: they’re shadowing themselves and the shadow volume is incorrectly shadowing them.
I suggest to not use stencil shadows, and to use some kind of shadow mapping. This is a really good tutorial on Variance Shadow Mapping: http://http.developer.nvidia.com/GPUGems3/gpugems3_ch08.html
son of a bitch, soft edges? I hope this doesn’t eat up too many resources.
/le sigh. time to redo shadowing code.
I implemented a cheap hack. Objects that cast shadows don’t receive them and objects that don’t cast shadows do receive them. (Scenery receives shadows, doesn’t cast, characters and such cast shadows, don’t receive)