Weird shadowmap problem

I am just implementing shadow mapping in my project, and I am having a weird problem: With a depth buffer FBO setup for shadow mapping (see below) nothing seems to get rendered to the shadowmap. When rendering the scene to an FBO with a regular depth buffer (color write masked off) and a color buffer, the depth buffer looks ok. I have changed the output so that the shadowmap gets rendered to the screen. Here’s what it looks like with the regular FBO:

When rendering the shadowmap depth texture, the entire output window is white (i.e. the shadowmap only contains 1.0, or what?)

Here’s the code for the shadowmap depth buffer:


GLuint hFBO, hDepth;
glGenFramebuffersEXT (1, &hFBO);
glBindFramebufferEXT (GL_FRAMEBUFFER_EXT, hFBO);
glActiveTexture (nTMU);
glEnable (GL_TEXTURE_2D);
glGenTextures (1, &hDepth);
glBindTexture (GL_TEXTURE_2D, hDepth);
// setup the depth texture
glTexParameteri (GL_TEXTURE_2D, GL_GENERATE_MIPMAP, GL_FALSE);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameterf (GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameterf (GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_COMPARE_FUNC_ARB, GL_LEQUAL);
glTexParameteri (GL_TEXTURE_2D, GL_DEPTH_TEXTURE_MODE, GL_INTENSITY); 				
glTexImage2D (GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT, nWidth, nHeight, 0, 
              GL_DEPTH_COMPONENT, GL_UNSIGNED_BYTE, 0);
glFramebufferTexture2DEXT (GL_FRAMEBUFFER_EXT, GL_DEPTH_ATTACHMENT_EXT, 
                           GL_TEXTURE_2D, hDepth, 0);

When rendering the depth buffer to screen I simply render it as a quad over the entire output window. Since it works for the regular depth buffer (24+8 bits depth+stencil), I guess it should work similary for the shadow texture.

Now the question is: Since the off screen renderer seems to work for a regular FBO render target, and the depth buffer looks fine there, why doesn’t it for the shadow texture? Is there something wrong in the way I am setting the shadow texture up?

See Example 7 in EXT_framebuffer_object.

In particular the glDrawBuffer/glReadBuffer part.

Funny thing is that I got that from a tutorial with a supposedly working shadow mapping implementation … I will remove it and see what happens.

Edit: Removing those two calls doesn’t help.

I have adjusted the code to call glDrawBuffer (GL_NONE) at the right place now, and have done more testing. Changing nothing but the way the depth buffer gets setup for rendering, it turns out that with the above code (shadow map) the depth buffer seems to remain unaffected during rendering. Setting it up as below, the depth buffer gets properly written to.


glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_COMPARE_FUNC_ARB, GL_LEQUAL);
glTexImage2D (GL_TEXTURE_2D, 0, GL_DEPTH24_STENCIL8_EXT, nWidth, nHeight, 
              0, GL_DEPTH_STENCIL_EXT, GL_UNSIGNED_INT_24_8_EXT, NULL);

Then I have removed the following line from the code in the first post:


glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE);

So if I set this texture property, the depth texture is completely white when rendered to screen. Why could that be?

What happens when you get the framebuffer status?

The frame buffer is reported to be complete. No errors. I am quering glCheckFramebufferStatusEXT (GL_FRAMEBUFFER_EXT) after creating the FBO.

Somehow that comparison mode doesn’t work right for me.

Your setup looks fine to me. I think the problem is elsewhere. Though I personally would be explicit and state GL_DEPTH_COMPONENT24 for the internal format.

Try this: don’t render anything to the depth texture. Just use glClearDepth( 0 ) and glClear( GL_DEPTH_BUFFER_BIT ) for forceably set all of the depth texture values to 0. Now do your debug render and verify you get black.

Now repeat this test but use glClearDepth( 1 ) and glClear( GL_DEPTH_BUFFER_BIT ). Do your debug render and verify you get white.

Note that when you’re doing your debug render, you do not want the depth compare flag set on the texture. Just pass it in through a sampler2D (not a sampler2DShadow), and do a normal “texture2D/texture” lookup on it (not “shadow2D”, if you’re using GLSL 1.2 or earlier). Grab the actual 0…1 value from the texture and plot that.

Then once you’re convinced you are getting good data from your test harness, go from there.

What do you mean with depth compare flag: GL_TEXTURE_COMPARE_FUNC or GL_TEXTURE_COMPARE_MODE?

As I said: Everything works when I remove the following call:


glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE);

I tried your proposition to just clear the depth buffer with various values and render it using a simple shader that gets it passed as “uniform sampler2D shadow” and accesses its contents with “gl_FragColor = vec4 (vec3 (texture2D (shadow, gl_TexCoord [0].xy).r), 1.0)”. This does not work when setting the texture compare mode to GL_COMPARE_R_TO_TEXTURE with the above code line present, but does work when not setting the texture compare mode. I also tried the shader with the frame rendered to a regular FBO (color+depth+stencil), and it worked fine. So the shader is ok, the FBO setup is ok, it’s just that one texture parameter.

Here’s the entire shadow code (it’s simple enough):


// vertex shader
void main()
{
gl_Position = ftransform();
gl_FrontColor = gl_Color;
}

// fragment shader
uniform sampler2D shadow;
void main()
{
gl_FragColor = vec4 (vec3 (texture2D (shadow, gl_TexCoord [0].xy).r), 1.0);
}

It looks like the depth texture is not being correctly read from. I am absolutely clueless about this. Is it that I only have to set the texture compare mode when actually doing the shadow mapping in the shader, and not when creating the depth texture used as shadow map?

Btw., hardware is NVidia GTX 570 with 266.something drivers (installed the latest NVidia drivers for that card yesterday).

Problem solved. I must only set the texture compare mode prior to actually applying the shadow map.