Depth Peeling - probably a dumb mistake

I’m trying to get a second peel working with the following fragment shader:

#version 330 core

layout(location=0) out float layer0;

uniform sampler2D depth;

uniform int peel;

void main()
{
    vec2 XY;
    XY = gl_FragCoord.xy;
    //XY = XY*0.5+0.5;

    if (peel==0)
    {
        layer0 = gl_FragCoord.z;
    }
    else
    {
        float prev_depth = texture(depth, XY);

        if (prev_depth > gl_FragCoord.z)
        {
            discard;
            //layer0 = prev_depth;
            //layer0 = 1;
        }
        else
        {
            layer0 = gl_FragCoord.z;
        }
    }
}

If I comment out the discard then I just get the depth map from the camera, but with the discard there, the whole texture is full of depth==1.

I’m not sure what’s going on, but I’d very much appreciate it if someone could point out my mistake.

Outputs can’t be conditionally assigned. If you remove the discard, an unspecified value will be assigned to the depth output whenever that branch is taken.

The implementation only copies gl_FragCoord.z to the depth output if the shader never writes to it explicitly, in which case the depth output will always be set from gl_FragCoord.z.

ok, that makes sense, sort of. So if I change my fragment shader to the following, then the depth test should auto-discard the bits in front of the first peel?

#version 330 core

layout(location=0) out float layer0;

uniform sampler2D depth;

uniform int peel;

void main()
{
    vec2 XY;
    XY = gl_FragCoord.xy;
    XY = XY*0.5+0.5;

    if (peel==0)
    {
        layer0 = gl_FragCoord.z;
        gl_FragDepth = gl_FragCoord.z;
    }
    else
    {
        float prev_depth = texture(depth, XY);

        if (prev_depth >= (gl_FragCoord.z-0.001))
        {
            //discard;
            //layer0 = prev_depth;
            layer0 = 1;
            gl_FragDepth = 1;
        }
        else
        {
            layer0 = gl_FragCoord.z;
            gl_FragDepth = gl_FragCoord.z;
        }
    }
}

(This one is giving me very wierd behaviour, with almost half the depth texture at ~0.7, and the other half at ~1, with a straight line down the center of the screen. I’m obviously missing something fundamental here)

gl_FragCoord.xy is in window coordinates (pixels), not NDC (-1…+1). So for the first few pixels on the bottom row, it will be (0.5,0.5), (1.5,0.5), (2.5,0.5),…

Use something lie


        float prev_depth = texelFetch(depth, ivec2(floor(gl_FragCoord.xy)), 0);

or


        float prev_depth = texture(depth, gl_FragCoord.xy / textureSize(depth, 0));

THANK YOU

That fixed that problem, and now I have found another issue when I try to draw the peeled layers with their proper colours.

I’m using this code (some non-relevent parts ommitted) to draw the things at depth 0 to layer 0, depth 1 to layer 1, etc…

#version 330 core

layout(location=0) out vec4 layer0;
layout(location=1) out vec4 layer1;
layout(location=2) out vec4 layer2;
layout(location=3) out vec4 layer3;

uniform sampler2D Depth0;
uniform sampler2D Depth1;
uniform sampler2D Depth2;
uniform sampler2D Depth3;

in vec4 colour_ambient_frag;

bool tol(float a, float b)
{
    float toll = 0.01;
    float an = a-b;
    return (an > -toll && an < toll);
}

void main()
{
    vec4 output;

    output = colour_ambient_frag;

    // handle depth peeling
    float depth_0 = texture(Depth0, gl_FragCoord.xy / textureSize(Depth0, 0));
    float depth_1 = texture(Depth1, gl_FragCoord.xy / textureSize(Depth1, 0));
    float depth_2 = texture(Depth2, gl_FragCoord.xy / textureSize(Depth2, 0));
    float depth_3 = texture(Depth3, gl_FragCoord.xy / textureSize(Depth3, 0));

    gl_FragDepth = 1;
    if (tol(gl_FragCoord.z,depth_0))
    {
        layer0 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_1))
    {
        layer1 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_2))
    {
        layer2 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_3))
    {
        layer3 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
}

The issue is that the code seems to stop after the first if statement closes. (If I check for depth 2 first, I get depth 2 drawn correctly, but none of the others, etc).

Again, I’m probably missing something fundamental, but I can’t figure out what it is.

As I said in comment #2 above:

This isn’t limited to gl_FragDepth; it applies to any output. Unless the fragment shader executes a [var]discard[/var] statement, the implementation will copy the values of all of the output variables (in this case, layer0, layer1, layer2, layer3, and gl_FragDepth) to the current pixel in the corresponding buffer (colour buffers 0 through 3 plus the depth buffer). If the shader didn’t actually assign a value to a particular variable (due to the assignment occurring within a conditional branch which wasn’t taken), the value written to the pixel will be undefined. But it will be written.

If you aren’t using the fourth component of each output variable, you can enable blending and set the fourth component to 0.0 or 1.0 to control whether a given buffer is modified.

My algorithm doesn’t care what goes into those outputs when I don’t write to them, but to make you happy (and show what my problem actually seems to be) I’ve adjusted my main function to the following:

void main()
{
    vec4 output;

    output = colour_ambient_frag;

    // handle depth peeling
    float depth_0 = texture(Depth0, gl_FragCoord.xy / textureSize(Depth0, 0));
    float depth_1 = texture(Depth1, gl_FragCoord.xy / textureSize(Depth1, 0));
    float depth_2 = texture(Depth2, gl_FragCoord.xy / textureSize(Depth2, 0));
    float depth_3 = texture(Depth3, gl_FragCoord.xy / textureSize(Depth3, 0));

    gl_FragDepth = 1;
    layer0 = vec4(0,0,0,0);
    layer1 = vec4(0,0,0,0);
    layer2 = vec4(0,0,0,0);
    layer3 = vec4(0,0,0,0);
    if (tol(gl_FragCoord.z,depth_0))
    {
        layer0 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_1))
    {
        layer1 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_2))
    {
        layer2 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_3))
    {
        layer3 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
}

My issue is that layers 1-3 are coming out pure black, but if I comment out the if statement for layer0, then layer1 coumes out how I would expect, same for layer2 if I comment out the if blocks for layers 0 and 1. So the following code gives me the correct output for layer1:

void main()
{
    vec4 output;

    output = colour_ambient_frag;

    // handle depth peeling
    float depth_0 = texture(Depth0, gl_FragCoord.xy / textureSize(Depth0, 0));
    float depth_1 = texture(Depth1, gl_FragCoord.xy / textureSize(Depth1, 0));
    float depth_2 = texture(Depth2, gl_FragCoord.xy / textureSize(Depth2, 0));
    float depth_3 = texture(Depth3, gl_FragCoord.xy / textureSize(Depth3, 0));

    gl_FragDepth = 1;
    layer0 = vec4(0,0,0,0);
    layer1 = vec4(0,0,0,0);
    layer2 = vec4(0,0,0,0);
    layer3 = vec4(0,0,0,0);
    /*if (tol(gl_FragCoord.z,depth_0))
    {
        layer0 = output;
        gl_FragDepth = gl_FragCoord.z;
    }*/
    if (tol(gl_FragCoord.z,depth_1))
    {
        layer1 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_2))
    {
        layer2 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
    if (tol(gl_FragCoord.z,depth_3))
    {
        layer3 = output;
        gl_FragDepth = gl_FragCoord.z;
    }
}

What am I missing?

That indicates that the test always passes.

Are you using an extremely small value for the near plane distance? That will result in most of the scene receiving a tiny fraction of the depth range, meaning that all depth values not very close to the near plane could be equal to within your 0.01 tolerance. As a rough guide, the range of -Z values beyond N*zNear receives the last 1/N of the depth range. So if zNear is 0.01 and your -Z values are greater than 1, the resulting depth values will be between 0.99 and 1.

More generally, depth peeling should only require inequality tests. Those tests shouldn’t require more tolerance than the precision of the depth buffer (6e-8 for a 24-bit depth buffer, 1.6e-5 for 16-bit), and in my experience don’t require any (i.e. using “if (gl_FragCoord.z <= oldz) discard;” is sufficient).

That would make sense if there was a single “else” statement in the shader, but there isn’t, so it should write to each layer that it’s close enough to.

I also tried changing my tolerence function to “a>=b”, and get the same behaviour - the comparison fails on everything except the first test.

Okay, a couple of things to try:.

[ul]
[li] Don’t assign the vec4 returned from texture() to a float. Use e.g. texture(…).r instead. Assigning a vec4 to a float should result in a compilation error.
[/li][li] Don’t assign outputs within conditionals. Create normal variables, initialise them, conditionally assign to those, then unconditionally assign them to the outputs at the end of the shader. This shouldn’t be necessary, but it seems less likely to trigger any implementation bugs.
[/li][/ul]

ok, tried both of those, same issue.

Current shader is this:

void main()
{
    vec4 output;

    output = colour_ambient_frag;

    // handle depth peeling
    float depth_0 = texture(Depth0, gl_FragCoord.xy / textureSize(Depth0, 0)).r;
    float depth_1 = texture(Depth1, gl_FragCoord.xy / textureSize(Depth1, 0)).r;
    float depth_2 = texture(Depth2, gl_FragCoord.xy / textureSize(Depth2, 0)).r;
    float depth_3 = texture(Depth3, gl_FragCoord.xy / textureSize(Depth3, 0)).r;

    float d = gl_FragCoord.z;
    float depth = 1;
    vec4 l0 = vec4(0,0,0,0);
    vec4 l1 = vec4(0,0,0,0);
    vec4 l2 = vec4(0,0,0,0);
    vec4 l3 = vec4(0,0,0,0);

    if (tol(d,depth_0))
    {
        l0 = output;
        depth = d;
    }
    if (tol(d,depth_1))
    {
        l1 = output;
        depth = d;
    }
    if (tol(d,depth_2))
    {
        l2 = output;
        depth = d;
    }
    if (tol(d,depth_3))
    {
        l3 = output;
        depth = d;
    }

    gl_FragDepth = depth;
    layer0 = l0;
    layer1 = l1;
    layer2 = l2;
    layer3 = l3;
}

Commenting out a conditional still affects values which should be dictated by subsequent conditionals?

Is the client code which sets up the program state affected by the absence of uniforms? As the values read from the textures are only used in conditionals, commenting out a conditional is likely to result in the corresponding sampler2D uniform being omitted from the program. Does forcing all 4 depth textures to be read (by making each value have an effect upon the output) have any effect?

I can’t see anything with the shader itself which could cause problems for anything resembling a functioning implementation, so I’d look for issues in the client code.