I’m working on some project with a group, and I’m trying to cast a shadow with the help of the stencil buffer. However I get a few weird effects since I installed my new video-card yesterday (a Radeon card). My previous GeForce didn’t show this side effect, and I can’t really think of any (big) changes I have made to the code since.

This is how it’s supposed to be:
How it’s meant to be

And this is what happens when I REBUILD the shadow

I hope the screenies show the problem clearly enough, the shadow is white.
What happens:
When the camera moved in my application, the shadow is rebuilt. That means the stencil buffer cleared, matrix for the projection calculated, and the objects drawn to the stencil buffer.
Then, if the camera hasn’t moved, the floor is drawn with a black-half-transparent color.
When the buffer is cleared, shadow rebuilt, the shadow lights up as white.

Some code that accompanies the shadow/floor drawing

{

//following vector and scalar are the plane in which the shadow lies
Vector3 N; //normal of the floor
float D; //distance to origin

Vector3 Q; //position of the light source

N.x = 0;
N.y = 1;
N.z = 0;
D = 3.3f;

Q.x = 0;
Q.y = 5;
Q.z = 15;

float dot = N.x * Q.x + N.y * Q.y + N.z * Q.z + D * 1;
float matrix[4][4];

glColor4f(0,0,0,0.6f);
glPushMatrix();

glEnable(GL_STENCIL_TEST);

{

``````  glClear(GL_STENCIL_BUFFER_BIT);	//maak de buffer leeg

//bereken de matrix
matrix[0][0] = dot - Q.x * N.x;
matrix[1][0] = 0.f - Q.x * N.y;
matrix[2][0] = 0.f - Q.x * N.z;
matrix[3][0] = 0.f - Q.x * D;

matrix[0][1] = 0.f - Q.y * N.x;
matrix[1][1] = dot - Q.y * N.y;
matrix[2][1] = 0.f - Q.y * N.z;
matrix[3][1] = 0.f - Q.y * D;

matrix[0][2] = 0.f - Q.z * N.x;
matrix[1][2] = 0.f - Q.z * N.y;
matrix[2][2] = dot - Q.z * N.z;
matrix[3][2] = 0.f - Q.z * D;

matrix[0][3] = 0.f - 1 * N.x;
matrix[1][3] = 0.f - 1 * N.y;
matrix[2][3] = 0.f - 1 * N.z;
matrix[3][3] = dot - 1 * D;

glMultMatrixf((float *)matrix);	//vemenigvuldig de nieuwe matrix met de ouwe matrix

if (glGetError())
MSGBOX("ERR");

glStencilFunc(GL_ALWAYS, 1, 0xFFFFFFFF);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

glBegin(GL_TRIANGLES);
glVertex3f(0,10,0);
glVertex3f(10,10,0);
glVertex3f(5,0,0);
glEnd();
//this->drawObjects(DRAW_OPTION_NO_COLOR);
``````

}
glStencilFunc(GL_EQUAL, 1, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

``````  	this->drawFloor(DRAW_OPTION_NO_COLOR);

glDisable(GL_STENCIL_TEST);
``````

glPopMatrix();
}

void CRender::drawFloor(int option)
{
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_NORMAL_ARRAY);
if (option != DRAW_OPTION_NO_COLOR)
{
glEnableClientState(GL_COLOR_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glColorPointer(4, GL_FLOAT, 0, floorColors);
glTexCoordPointer(2, GL_FLOAT,0, floorTexCoords);
}

glVertexPointer(3, GL_FLOAT, 0, floorVertices);
glNormalPointer(GL_FLOAT, 0, floorNormals);

``````glDrawArrays(GL_QUADS, 0, DRAW_FLOOR_SIZE * DRAW_FLOOR_SIZE * 4);
``````

glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_NORMAL_ARRAY);
glDisableClientState(GL_COLOR_ARRAY);
glDisableClientState(GL_TEXTURE_COORD_ARRAY);

}

I have tried to track down the problem, and so far it SEEMS as if glMultMatrix causes this. If I comment out the glMultMatrix in the drawShadow function, it uses the correct color. Else it is white, though it does blend correctly.
Is there any sideeffect to glMultMatrix, it resetting something?
Anything I might have overlooked? The clearcolor perhaps? I’ve tried a lot of things, and can’t find it.
I hope someone can give me a pointer to how I can fix this.

Thanks for your help, and have a happy new year

[This message has been edited by Structural (edited 12-31-2003).]

Is there any chance you’re working on the color matrix instead of the modelview matrix at this point? If you call glMatrixMode( GL_MODELVIEW ) prior to your matrix ops does it start working?

I can’t imagine how anything would work correctly if this were the case, but it’s the first thing that comes to mind.

– Jeff