a few days ago i converted my engine to work on the ati 9600 pro and i found that when i tried to overwrite fragments using glDepthFunc(GL_EQUAL) strange and huge stencil artifacts aappeared on the screen, is this a driver issue?
what i’m doing is this:
-fill zbuffer with scene depth
-cast stencil shadows
-draw scene (using glDepthFunc(GL_EQUAL) on non-shadowed areas
the result is that on non-shadowed areas there are certain stencil artifacts that appear in some kind of strange patterns.
i had my engine working perfectly on nvidia boards so i think this error is just a driver bug.
one more thing, i’m sorry for posting about ati issues. i know this forum has a lot of them.
I’ll bet these are not stencil artifacts. These are probably zbuffer invariance problems. Your depth fragments for the lighting pass do not match the depth fragments for the depth fill pass, so some pixels pass and others fail the depth test and you either see lighting or no lighting in a random shimmering pattern depending on the modelview matrix contents.
This invariance may be caused by differing OpenGL state between passes or fragment shaders if you use them. Please supply more details on the state applied during the zfill and the illumination passes (do you use shaders?). Different cards have different invarince behaviour, that doesn’t mean the problem is with ATI, it could still be your code and you just got lucky with NVIDIA’s implementation. Even with this working on one platform less obvious things can hit you later, like clipped primitives or primitives that lie outside the guard band can reintroduce invariance if you’re not careful, so it always pays to get this right.
First, try using LEQUAL when drawing color, and see if it fixes the problem.
If that’s not good enough, i e the second pass generates depth coordinates further into the scene than the first pass, then use LEQUAL, and move the near clipping plane a little bit out from the viewport; something like 0.001 times the previous value should be plenty, while not leading to much aliasing.
Or use a single point of polygon offset, still with LEQUAL in the second pass.