Stencil Question

My aim is to determine a sequence of (stencil) operations so that regardless of the order in which objects are rendered I can end up with the visible (non-occluded) part of any object (believe that I am looking at rendering objects that pass the stencil test but do not pass the depth test).

I have the following code:

        glStencilMask(0xFF)
        glStencilFunc(GL_ALWAYS, 1, 0xFF) # All fragments will pass
        glStencilOp(GL_KEEP, GL_REPLACE, GL_KEEP)
        
        # cube 1. Large cube that I want to render to the stencil
        glBindVertexArray(cubeVAO)
        glActiveTexture(GL_TEXTURE0)
        glBindTexture(GL_TEXTURE_2D, cubeTexture)
        model = glm.mat4(1.0)
        model = glm.translate(model, glm.vec3(0.0, 0.0, -0.75))
        shader.setMat4("model", model)
        glDrawArrays(GL_TRIANGLES, 0, 36)
        
        # Stop rendering to the stencil
        glStencilMask(0x00) 

        # cube 2
        scale = 0.35
        model = glm.mat4(1.0)
        model = glm.scale(model, glm.vec3(scale, scale, scale))
        model = glm.translate(model, glm.vec3(0.0, 0, 2.0))
        shader.setMat4("model", model)
        glDrawArrays(GL_TRIANGLES, 0, 36)
        glBindVertexArray(0)

Here are two renderings. The one on the left shows distance to the two cubes (and a floor). The one to the right shows what is the result of the stencil operations above.

image

The small cube that we can see on the left is clearly in front of the large cube yet the output I get on the right does not seem to correspond at all with what I would expect based on the left image. I am using pyOpengl and I am reading the results from the stencil buffer into a numpy array (for those that are familiar with these). I use the following code to do this (I am including it here in case I am inadvertendly changing something),

stencil = glReadPixels(0,0, SCR_WIDTH, SCR_HEIGHT, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE)
stencil = np.fliplr(np.frombuffer(stencil, dtype=np.uint8).reshape((800,-1)))

I am trying to figure out this disparity.

I have found a tiny mistake when I was copying the stencil buffer onto a numpy array. I had

stencil = np.fliplr(np.frombuffer(stencil, dtype=np.uint8).reshape((800,-1)))

when instead I should have had

stencil = np.frombuffer(stencil, dtype=np.uint8).reshape((800,-1))

this still does not explain why the occlusion of the smaller cube (right) does not match what appears when mapping the distance (left)
image

I have not been able to figure this out yet. There is definitely a disconnect between what I am rendering and the result I am getting from the stencil buffer. I changed the size of the cube that is supposed to be blocking the larger cube and I get exactly the exact same result from the stencil buffer.
image

Any ideas on what I might be doing wrong welcome!

Still stuck on this! I cannot say whether my thinking is totally off or something else is going on!!!

I am just including part of the code that directly concerns what I am trying to resolve. If I type the following

    glClearColor(0.1, 0.1, 0.1, 1.0)
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT) 

    # Setting the stencil buffer
    glStencilMask(0xFF) # where am I writing
    glStencilFunc(GL_ALWAYS, 1, 0xFF) # All fragments will pass
    glStencilOp(GL_KEEP,    # stencil fails
                GL_KEEP,    # stencil passes, depth fails
                GL_REPLACE)    # stencil and depth passes
    
    # cube 1
    glBindVertexArray(cubeVAO)
    model = glm.mat4(1.0)
    model = glm.translate(model, glm.vec3(0.0, 0.0, -0.75))
    shader.setMat4("model", model)
    glDrawArrays(GL_TRIANGLES, 0, 36)
  
    # set it so that we do not draw the next to the stencil
    glStencilMask(0x00)

    # cube 2
    scale = 0.25
    model = glm.mat4(1.0)
    model = glm.scale(model, glm.vec3(scale, scale, scale))
    model = glm.translate(model, glm.vec3(0.0, 0, 2.0))
    shader.setMat4("model", model)
    glDrawArrays(GL_TRIANGLES, 0, 36)
    glBindVertexArray(0)
           
    # OUTPUTS                  
    #depth
    depth = glReadPixelsf(0,0, SCR_WIDTH, SCR_HEIGHT, GL_RED, GL_FLOAT)
    np.save('depth', np.flipud(depth))
    
    # stencil
    stencil = glReadPixels(0,0, SCR_WIDTH, SCR_HEIGHT, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE)
    stencil = np.frombuffer(stencil, dtype=np.uint8).reshape((800,-1))
    np.save('stencil', np.flipud(stencil))

I get the following result as expected,
image

That is the stencil buffer is being replace by the reference value 1 for all fragments that pass the stencil and depth test. Now, if from the above code I just change the following line,

    glStencilOp(GL_KEEP,    # stencil fails 
                GL_REPLACE,    # stencil passes, depth fails
                GL_KEEP)    # stencil and depth passes

I get the following result,
image
Which does not correspond at all to what I would have expected. The only thing I can think of is that I am reading incorrectly the stencil buffer (but even that would not make much sense).

Ok I managed to get the result I was after but I still cannot explain why this was not happening with my previous code so if anyone can point out where my thinking was wrong or why my code was incorrect it would be greatly appreciated!!!

To achieve the result I wanted from the start I used the following code,

    glStencilMask(0xFF) # where am I writing
    glStencilFunc(GL_ALWAYS, 1, 0xFF) # All fragments will pass
    glStencilOp(GL_KEEP,    # stencil fails
                GL_KEEP,    # stencil passes, depth fails
                GL_REPLACE)    # stencil and depth passes

    # cube 1 (large cube)
    glBindVertexArray(cubeVAO)
    model = glm.mat4(1.0)
    model = glm.translate(model, glm.vec3(0.0, 0.0, -0.75))
    shader.setMat4("model", model)
    glDrawArrays(GL_TRIANGLES, 0, 36)
    
    glStencilOp(GL_KEEP,    # stencil fails
                GL_KEEP,    # stencil passes, depth fails
                GL_ZERO)    # stencil and depth passes
    
    # cube 2 (small cube)
    scale = 0.25
    model = glm.mat4(1.0)
    model = glm.scale(model, glm.vec3(scale, scale, scale))
    model = glm.translate(model, glm.vec3(0.0, 0, 2.0))
    shader.setMat4("model", model)
    glDrawArrays(GL_TRIANGLES, 0, 36)
    glBindVertexArray(0)

Main differences: the entire large cube, not just the visible part as I originally wanted, is first stored in the stencil buffer. Then the second smaller cube is also used to update the stencil buffer BUT instead of using a value of 1 as I did for the larger cube I used 0. This solution works (see below) but it is not ideal
image

The question remains, how to store only the visible parts of an object in the stencil?

Forget code for a minute. Let’s define what you want.

What do you mean by: “so that … I can end up with the visible (non-occluded) part of any object”? In what buffer? STENCIL? I’m assuming yes, because if you meant COLOR + DEPTH, you can get that without messing with STENCIL. So…

What specific values do you want in the STENCIL buffer for:

  1. Pixels (**) where “no” object is written?
  2. Pixels where exactly 1 occluder is written?
  3. Pixels where multiple occluders may be written and only the closest was kept (due to the depth test)?

(**) - or samples, if MSAA enabled.

Strawman proposal: Clear color+depth+stencil. Set stencil to write a “1” for every fragment rasterized. Then perform object rendering in the standard way, with depth test and depth writes ON and additionally with stencil writes on. What about this does not give you what you want?

Hi @Dark_Photon

Apologies for the delay. Thanks for taking this up!

I am dealing only with the stencil buffer.

So what I want is to get the pixels that are still visible (not the ones that are occluded) associated with an object after removing those that are occluded by any other objects that are in front (closer to the viewer).

For instance, a trivial example, I have a house with a tree in front of it. I want the bits of the house that are not blocked by the tree (i.e. that are visible)

I thought this could be accomplished via stencil testing in the following way:

  1. Clear color+depth+stencil
  2. Set stencil mask to 255
  3. Set the stencil test to ALWAYS pass, use as reference value 1 and function mask 255
  4. Set the stencil operation to replace the stencil buffer value with the reference value whenever it passes the stencil test but fails the depth test
  5. Draw a large cube (or following my example a house)
  6. Set the stencil mask to 0 (so nothing passes the stencil test)
  7. Draw a small cube in front of the large cube (or draw the tree), that is closer to the viewer.

What I was expecting were the bits of the larger cube minus the bits blocked by the smaller cube (the house bits that are visible). But I do not get this. I cannot see why this is not so (not at least from what I understand how the stencil ops should work).

I hope this clarifies at least what I am after.

Thanks again!

I’m already trying to answer you on gamedev.net (and provided you with a working solution).

Since this is less a monologue on the other website, you might consider continue there.

Thanks @Silence for the sake of the community I want to include what you suggested in our conversations. In case anyone is interested. Your suggestion was posted here:

  1. Render your scene normally, without the object you want to test
  2. Enable stencil, clear stencil with 0, set stencil op to keep, keep, replace, and stencil func to (always,1,1)
  3. Draw the object you need its visible pixels

You should then have the stencil buffer filled with 1 where your object is visible, and 0 everywhere else

This works and I get the result I was after. I am still trying to understand why what I suggested above…

  1. Clear color+depth+stencil
  2. Set stencil mask to 255
  3. Set the stencil test to ALWAYS pass, use as reference value 1 and function mask 255
  4. Set the stencil operation to replace the stencil buffer value with the reference value whenever it passes the stencil test but fails the depth test
  5. Draw a large cube (or following my example a house)
  6. Set the stencil mask to 0 (so nothing passes the stencil test)
  7. Draw a small cube in front of the large cube (or draw the tree), that is closer to the viewer.

…does not work.

I’m into trying to make you understand. Hopefully my last answer is clearer.

Hi @Silence I think I understand, at least one reason, why my thinking was incorrect.