OpenGL Shadows and the Stencil Buffer

OpenGL OpenGL: Basic Coding
1

So, I’ve been following nehe’s guide to creating shadows, and for whatever reason the shadows just aren’t rendering. Not having an excellent understanding of things like the stencil-buffer means I’m sort of learning things as I go along. I hate outsourcing my debugging, but I’m kind of stumped here. does anybody know what’s wrong?


glEnable(GL_CULL_FACE);
glCullFace(GL_FRONT);
glPushAttrib( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_ENABLE_BIT | GL_POLYGON_BIT | GL_STENCIL_BUFFER_BIT );
glDisable( GL_LIGHTING );
glDepthMask( GL_FALSE );
glDepthFunc( GL_LEQUAL );
glEnable( GL_STENCIL_TEST );
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFFL );

glFrontFace( GL_CCW );
glStencilOp( GL_KEEP, GL_KEEP, GL_INCR );

//shadow volume quad drawing code here
//I've checked to make sure that the code to draw the volume is running
//so I thought I'd spare you that

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

glColor4f(0.0f, 0.0f, 0.0f, 0.4f);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glStencilFunc(GL_NOTEQUAL, 0, 0xffffffff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

glPushMatrix();
glLoadIdentity();
glMatrixMode(GL_PROJECTION);
glOrtho(0, 1, 1, 0, 0, 1);
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();

glBegin(GL_QUADS);
	glVertex2i(0, 0);
	glVertex2i(0, 1);
	glVertex2i(1, 1);
	glVertex2i(1, 0);
glEnd();

glEnable(GL_DEPTH_TEST);
glPopMatrix();
glPopAttrib();

glEnable(GL_LIGHTING);
glDisable(GL_STENCIL_TEST);
glShadeModel(GL_SMOOTH);
glDisable(GL_CULL_FACE);

And I run that every frame. I have a feeling it’s a problem in how I arranged the various gl*() commands. Only the large, screen-covering shadow is drawing with no parts being cut out.

2

Hi there. I wanted to find out how to do shadows too and discovered how to do it with what’s called a shadow matrix. I can give you the relevant code and at least partly explain how it works. I will admit though that it uses math from linear algebra which kind of above my head. I never took that course in college.

3

I’m going to take that course eventually, but yes I’d love to see it. I don’t know any linear algebra but perhaps I can compare that to my code and see what’s going on, or use it instead or something.

4

I can help you but its first time I am using this page. give me you problem exactly I will help you
Hoshang Kolivand
shahinkey@yahoo.com

5

Ok, put these two functions above your drawScene function:

void calcNormal(float v[3][3], float out[3])
{
float v1[3], v2[3], length;
static const int x = 0, y = 1, z = 2;

v1[x] = v[0][x] - v[1][x];
v1[y] = v[0][y] - v[1][y];
v1[z] = v[0][z] - v[1][z];

v2[x] = v[1][x] - v[2][x];
v2[y] = v[1][y] - v[2][y];
v2[z] = v[1][z] - v[2][z];

out[x] = (v1[y] * v2[z]) - (v1[z] * v2[y]);
out[y] = (v1[z] * v2[x]) - (v1[x] * v2[z]);
out[z] = (v1[x] * v2[y]) - (v1[y] * v2[x]);

length = (float)sqrt( (out[x] * out[x]) + (out[y] * out[y]) + (out[z] * out[z]) );

out[x] = out[x] / length;
out[y] = out[y] / length;
out[z] = out[z] / length;
}
void MakeShadowMatrix(GLfloat points[3][3], GLfloat lightPos[4], GLfloat destMat[4][4])
{
GLfloat planeCoeff[4];
GLfloat dot;

calcNormal(points, planeCoeff);

planeCoeff[3] = - ( (planeCoeff[0] * points[2][0]) + (planeCoeff[1] * points[2][1]) + (planeCoeff[2] * points[2][2]) );

dot = (planeCoeff[0] * lightPos[0]) + (planeCoeff[1] * lightPos[1]) + (planeCoeff[2] * lightPos[2]) + (planeCoeff[3] * lightPos[3]);

destMat[0][0] = dot - (lightPos[0] * planeCoeff[0]);
destMat[1][0] = 0.0f - (lightPos[0] * planeCoeff[1]);
destMat[2][0] = 0.0f - (lightPos[0] * planeCoeff[2]);
destMat[3][0] = 0.0f - (lightPos[0] * planeCoeff[3]);

destMat[0][1] = 0.0f - (lightPos[1] * planeCoeff[0]);
destMat[1][1] = dot - (lightPos[1] * planeCoeff[1]);
destMat[2][1] = 0.0f - (lightPos[1] * planeCoeff[2]);
destMat[3][1] = 0.0f - (lightPos[1] * planeCoeff[3]);

destMat[0][2] = 0.0f - (lightPos[2] * planeCoeff[0]);
destMat[1][2] = 0.0f - (lightPos[2] * planeCoeff[1]);
destMat[2][2] = dot - (lightPos[2] * planeCoeff[2]);
destMat[3][2] = 0.0f - (lightPos[2] * planeCoeff[3]);

destMat[0][3] = 0.0f - (lightPos[3] * planeCoeff[0]);
destMat[1][3] = 0.0f - (lightPos[3] * planeCoeff[1]);
destMat[2][3] = 0.0f - (lightPos[3] * planeCoeff[2]);
destMat[3][3] = dot - (lightPos[3] * planeCoeff[3]);
}

Now declare these at begining of drawScene function:

GLfloat shadowMat[4][4];
GLfloat points[3][3];

Set points to three points that lie on the plane where the shadow will be cast

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); //don’t forget this

And then when you’re ready to draw a shadow put this code before it with a light source of your choosing:

MakeShadowMatrix(points, lightPos0, shadowMat);

glPushMatrix();

glMultMatrixf((GLfloat *)shadowMat);

glColor3f(0.0f, 0.0f, 0.0f);

//draw the thing casting the shadow again    

glColor3f(1.0f, 1.0f, 1.0f);

glPopMatrix();
6

Set points to three points that lie on the plane where the shadow will be cast

Does the code rely on the fact that the shadow will be drawn on a single plane? Because it won’t be.

7

Umm, good question. I’m really not sure. Sorry. I thought I could be of help.

8

Lol. where’d you get the code then?

9

OpenGL Super Bible Second Edition. Found the book at the main library. Its been incredibly helpful so far. I’m new to OpenGL. :slight_smile:

10

Holy balls I need that book. Is there any code for projecting shadows onto something other than a plane?

11

I’d have to dive into it again. However its 2am here right now and I’m beat. Gonna hit the sack. I’ll follow up with you tomorrow.

12

Alright, see you tomorrow, and thanks for the help. Hopefully this gets resolved because I’ve been debugging for a week and want to move on to particles and reflections.

I’m eventually going to be making a game based on the Israeli war of independence. maybe.

13

Yes it’s called shadow mapping. There are loads of examples on the net and you’ll find this technique better than planar stencil shadowing.
I also suggest you you shaders as this greatly simplifies the process of supplying the light projection * inverse scene camera matrix as opposed to using glTexgen commands.

14

thanks, I’ll take a look at that (although I wasn’t using glTexgen commands to begin with for shadows)

15

You will do with shadow mapping because you need to compare the light source depth buffer (aka shadowmap) with the current cameras depth buffer. To do that you need to supply texture coordinates to index the shadowmap and to do that you need to send in the matrix consisting of lights projection matrix * lights camera * inv scene camera.

16

Why should I do shadow mapping when a) I’ve already hacked out most of the shadow volume rendering and b) it’s a bit less accurate?

17

Well, on NeHe’s site, it says:

So I’d suggest you drop back and try some simple stencil stuff with a 2D quad to get familiar with it. That’s what I did when I was learning about it.

Before you beat your head against it too much more though, what are you shadow requirements? How many shadow casting objects? How many faces per object? Are they convex? Are they guaranteed to have no cracks? Can a shadow volume potentially intersect the near plane inside the frustum? How accurate do your shadows have to be? Do you need soft shadows? Are your lights and/or objects dynamic? Do you have area lights, or just point and directional lights? How big is your “world”? Do you need translucent or “fuzzy” objects (like volumetric smoke) to cast a shadow? What kind of CPU/GPU are bringing to the table? What res are you doing this at? Do you need sub-pixel accurate shadows? What frame rate do you have to maintain? etc.

You should be aware that that NeHe article is kinda old as shadow volumes tech goes, only describes a small subset of the basics of shadow volumes, and (as it even says) builds and renders shadow volumes very inefficiently. It describes Z-Pass shadow volumes (without calling them that), but doesn’t detail the failure cases or what to do about them, and the perf tradeoffs of those options. And it doesn’t talk about building shadow volumes more efficiently (the ancient Gamasutra article it points to for that doesn’t do a whole lot better by modern tech standards, because it’s 12 years old!)

Why should I do shadow mapping when a) I’ve already hacked out most of the shadow volume rendering and b) it’s a bit less accurate?

It’s all a matter of what your requirements are. Depending on your perf and quality requirements (and your graphics tech know-how at this point in your learning), shadow volumes might be just perfect for you and your needs, …or they might be so completely wrong that it’s not even funny.

Give us some more info about your requirements and we can help point you toward the path of most likely success.

18

Sure. The end goal is to make a video game, although I recognize that’s quite a long way off. However, the engine should be made with that in mind.

So, there will likely be a few hundred faces per object, convex and concave, often with cracks, and the shadows could easily reach the near plane. My shadows should preferably accurate to the pixel, and the objects will be dynamic though the light probably won’t be. I will have a few light sources at any given time, mostly area lights but directional lights aren’t out of the question. The world will be reasonably extensive: it certainly won’t fit on just one or two screens. Translucent objects will probably cast a shadow (haven’t decided), but I don’t think I’ll bother with particles and volumetric stuffs. For the really low end computers I’ll just enable “circle under the model” shadows, but I expect anybody with something at the level of a 9800+ to be able to render shadows, although I’d prefer it to be less than that. As a game, this should be able to run at up to 1920 x 1080. They don’t need to be sub-pixel accurate, but for the aforementioned hardware I’d like to maintain 30+ FPS.

If shadow volume rendering is an idiotic way to go then please tell me. I just want whatever works.

19

Okay. So I forgot to tell glut to enable the stencil buffer. Luckily, your time isn’t wasted because I’m still left with problems.

No shadows render. If I allow writing to the color buffer, then I can see that the shadow volume is rendering and that it’s doing so correctly. However, it’s not writing to the stencil buffer properly as, when I disable rendering to the color buffer, no shadows appear.

20

And I got it. Well, not quite there are still some odd artifacts, but basically I got it.