Not satisfied with stencil shadows

I implemented stencil shadows, but I am not really happy with them. Shadows go through solid objects, and appear on surfaces that face away from the light source. They just don’t seem like a good technique for real-time programs.

I am using lightmaps for brush solids, hardware lights to illuminate moving objects, and vertex colors for static meshes. Stencil shadows are used on movable physical objects. There are so many errors that can occur if you cast shadows on walls, that it makes everything look really unrealistic. If I go outside the building shown below, there will be shadows cast onto the ground outside, coming out of the building.

What can I do to improve these for real-time use? I thought about just casting shadow downwards away from the light source, just to have simple ground shadows.

[img]http://www.leadwerks.com/post/stencils6.jpg[/img]

Originally posted by halo:
I implemented stencil shadows, but I am not really happy with them. Shadows go through solid objects, and appear on surfaces that face away from the light source. They just don’t seem like a good technique for real-time programs.
That’s because you see shadows as something you add to the scene, what you should be doing is to add light except where there are no shadows.
A bit poetic, but when it comes to light and shadows you need to begin at the right end

Originally posted by halo:
[b] I am using lightmaps for brush solids, hardware lights to illuminate moving objects, and vertex colors for static meshes. Stencil shadows are used on movable physical objects. There are so many errors that can occur if you cast shadows on walls, that it makes everything look really unrealistic. If I go outside the building shown below, there will be shadows cast onto the ground outside, coming out of the building.

What can I do to improve these for real-time use? I thought about just casting shadow downwards away from the light source, just to have simple ground shadows.
[/b]
When it comes to stencil shadows, you have to light everything with them (or at least with some other sort of dynamic lighting), or else it will look bad, the second image is a classic example of this, that image wouldn’t look so bad if the entire building would cast a shadow.
Remember, shadows are not some kind of magic rays that stops on contact with a surface, it’s the void of light and they will continue for ever.

I’m writing a large-scale FPS engine. I don’t want to be limited to little corridors. Is it possible to render large-scale terrains like in UT2004, using a system like what you are suggesting?

The worst case scenario is I can treat stencil shadows like projected shadows, and just cast them downwards, and limit them to about 3 feet.

Shadows go through solid objects…

These solid objects should cast shadows, too.

…and appear on surfaces that face away from the light source

Shadows only tell wchich part of the screen should receive lighting (from one light source). If some polygon faces away from light source then it makes no difference if it’s in shadow or not.
Your problem is that you have no proper lighting implemented. Shadowmaps and stencil shadows are techniques that are not stand-alone. They’re always part of lighting engine.
You should not darken the areas that receive shadows. You should start with black scene. Then for each light source you should cast shadows (into stencil buffer - not into framebuffer) and perform lighting operations on areas not in shadow . Lighting from each light source should be added to scene.

Originally posted by halo:
I’m writing a large-scale FPS engine. I don’t want to be limited to little corridors. Is it possible to render large-scale terrains like in UT2004, using a system like what you are suggesting?
Yes, ever heard of UT2007 or doom 3 or crysis.
Shure stencil shadows work best in cramped little corridors, but it can allso work fairly well in large outdoor terrains if you do it right.
Projected depthmap shadows works really well with large oudoor terrains and they do work well with
stencil shadows.

uhm… doors should cast shadows too. So… when you do this, second screenshot will look correct.

Yes, ever heard of UT2007 or doom 3 or crysis.
Doom only handles tiny areas, and the other two don’t exist yet. I don’t want to develop my engine based on what Epic says their engine will be able to do in two years.

OK, let’s finish this.

Halo (?), you’re doing your shadowing wrong. You’re thinking about shadowing wrong.

People (most recently k_szczech) have told you how to fix the problems you’re seeing. But you’re not listening to them. So, I will sum up what’s wrong with your implementation.

First, you’re shadowing through 2 different methods: light maps and stencil shadows. This is doable, but you have to think of shadowing correctly in order to make it work.

When you’re rendering a surface, there are now 2 ways for it to be “in shadow”. One is from the light map. The other is from the stencil shadow (which should be detected by the stencil pass, so you won’t see it in the fragment program). You must make certain that both of these will result in the emission of the same color.

So, one problem is that this isn’t the case. Your shadow maps never seem to go to full 0, but stencil shadows do seem to. That is because your ambient pass isn’t working correctly. You need to have an ambient pass that matches up with the ambient color from the light map, so that your light map can actually go to 0. So, really, you need two light maps; one is for the ambient, and the other is for the regular diffuse pass.

Next, you’re not shadowing everything everywhere. Your outdoor shot with the brick wall is a prime example. There are clearly two light sources in the scene. The building is in the shadow of one of them, but there is no evidence of this from the light map. The results of the stencil shadow say that one of the objects being stencil shadowed is in fact in the shadow of that light.

It’s pretty much up to you to fix this. The principle problem is that you are not shadowing evenly; not everything casts shadows and the shadows are not being handled correctly when the two methods overlap.

That’s your problem.

I understand how to do multiple lighting passes just fine. I was hoping to avoid such a system, because I find them to be extremely limited in terms of what kind of environments they can display. I was hoping there might be some other tricks I had not thought of, without going the whole Doom 3 route, and having to render my whole scene 3 times extra.

This screenshot from Epic clearly shows that their engine is still using lightmaps in some areas:

Another example comes from the Far-Cry engine site:

Lighting and Shadows: Combines pre-calculated, real-time shadows, stencil-shadows, and lightmaps to produce a dynamic environment. Includes high-resolution, correct-perspective, and volumetric smooth-shadow implementations for dramatic and realistic indoor shadowing.
F.E.A.R. also combined stencil shadows with lightmaps.

Originally posted by halo:
I understand how to do multiple lighting passes just fine. I was hoping to avoid such a system
With stencil shadows you can only do one light at a time. If you want to draw light from multiple light sources in a single pass, you may want to consider shadow mapping.

What is shadow mapping? Do you just mean conventional lightmapping?

What about Far Cry and FEAR, which clearly use stencil shadows with lightmaps, and are able to limit the shadows so they don’t go through walls? In Fear, even the static lights cast stencil shadows around the meshes.

Shadow mapping renders the depth from the light’s point of view to a texture, then compares the stored depth to computed depth in the lighting pass. If depth is larger than stored depth, this mean something is occluding, which means you’re in shadow.
With stencil shadows you have only one stencil buffer, and stencil operations conditionally kill fragments, so it’s either in or out of shadow, no middle ground or ability to be in one shadow but out of another, except by doing multiple passes. That’s what Fear, Doom3 and all these games do. As for limiting so that shadows don’t go through walls, these games don’t do anything particular about that. It just works because the wall also casts a shadow, so it doesn’t matter that it goes through the walls, it will always be in shadow on the other side.

I changed the code a little to do multiple passes, with every object casting a shadow. Framerate is okay when no lights are moving, and I am only inside a small area. (The light is moving here).

What if I want to do scenes bigger than a little corridor? What kind of optimizations can I make just to get static shadow volumes running in a large scene?

Put a bounding scissor rectangle around your light. Don’t extrude the shadow volume longer than the light radius.

Yeah, I have tried everything.

Here is a stress-test. It’s not pretty, because I just converted a system over really fast, but it draws a crapload of static volumes:
http://www.leadwerks.com/post/supertest.zip

With 800 shadow volumes, a GEForce 7800 is only getting about 30 fps. I don’t think the hardware is ready to do what I want it to.

how many lights are in that test?
are you using the scissor test trick?
are you calculating silhouettes on cpu?
why are you using stencil volume shadows anyway? what’s your reasoning apart from ‘fear uses em’.

-Just one directional light.
-I am using a BSP brush that gets built around the static volume, and tested for intersection with the camera frustum, so it culls offscreen shadow volumes.
-I am calculating silhouettes in CPU, but since the light doesn’t move, they only get calculated once.
-Stencil shadows and volumetric lighting are a much more realistic way to handle lighting. However, their hardware demands set everything back quite a bit, and I am afraid they will be too limiting.

Quake Wars is the biggest scale of scenery I have ever seen with uniform lighting, and it looks like most of it is just a single directional light, with very little architectural detail. Are there any examples of large-scale environments running in real-time with uniform lighting?

so you’re not using the scissor test trick then - that’s very different from simple frustum culling.

stencil shadows and volumetric lighting are not the same thing, and I fail to see how stencil shadow volumes are ‘a much more realistic way to handle lighting’. They’re just another shadow approximation approach, and are neither the most realistic or efficient realtime method available on todays hardware.

Anyway, back to your question about shadow volumes passing through walls - it’s trivial when calculating your shadow volume to do a ray intersect with your world geometry to give you a plane at which to clip the volume.

The scissor test is only relevant to point and spot lights, not directional.

Clipping to the first face that gets hit would be easy, but would create a tremendous amount of visual glitches.