What I mean by “punch a hole into the depth/stencil image” is with a pipeline that has stencilTestEnable set to true, if you render with an image that is black with white spots, the white areas will allow other rendering to pass.
I just had a thought, does the image I want to render to the stencil buffer need it’s aspect mask set to VK_IMAGE_ASPECT_STENCIL_BIT? Right now the image I’m using as a stencil mask is visible in the render even though stencilTestEnable = true in that pipeline. Not sure what I’m doing wrong.
if you render with an image that is black with white spots, the white areas will allow other rendering to pass.
Two things. First, I don’t understand what you mean by “allow other rendering to pass”. Do you want previous rendered stuff to be visible through these white areas? Do you want it to affect subsequently rendered objects?
Second, the stencil buffer doesn’t care about colors. And since the fragment shader cannot manipulate the stencil buffer, you can’t do a thing where the FS detects “white” and causes something stencil-related to happen.
The stencil test is about math, operations done based on the stencil value for the fragment (constant for all fragments in the rendering command) and the stencil value in the corresponding sample in the stencil buffer. And some depth stuff too.
the image I want to render to the stencil buffer
You cannot render “colors” to the stencil. And, as previously stated, the fragment shader cannot affect the fragment’s stencil value. The stencil buffer’s value can only be affected by the stencil reference value you set when you rendered those objects. It’s a single, unchanging value over the course of the rendering command.
Do you want previous rendered stuff to be visible through these white areas? Do you want it to affect subsequently rendered objects?
Yes. In OpenGL, I would disable the depth and color buffers, enable the stencil buffer, this would now render to the stencil buffer, re-enable the color and depth buffers and what I rendered would only show through what I did to the stencil buffers. I know it’s all math but I’m a visual thinker and communicator. For my needs, I only used the stencil buffer for simple masking.
The stencil buffer’s value can only be affected by the stencil reference value you set when you rendered those objects.
Can you tell me where in the example I sited it’s doing that? Do you have any example code?
Then do that in Vulkan. It has all of the same switches as OpenGL does. Vulkan has write masks, just like OpenGL. Vulkan has the same stencil reference state, as well as stencil function state.
By “disable/enable the color and depth buffers”, I assume you mean turning off write masks. Because changing the FBO itself is highly unnecessary for such a task.
You said “if you render with an image that is black with white spots, the white areas will allow other rendering to pass.”. “White” and “black” are colors; ergo, you want to use colors to determine the value of the stencil. That’s not allowed.
So either you haven’t explained very well what you’re trying to do, or what you’re trying to do isn’t possible, in either OpenGL or Vulkan.
You said “if you render with an image that is black with white spots, the white areas will allow other rendering to pass.”
Here’s some of my OpenGL code using a stencil buffer to mask the items in a scroll box. Since a stencil is an 8 bit buffer, you can think of it as a grey scale image or perhaps something similar to an alpha channel.
// Disable color and depth buffers
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glDepthMask( GL_FALSE );
// Start using the stencil
glEnable( GL_STENCIL_TEST );
glStencilFunc( GL_ALWAYS, 0x1, 0x1 );
glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );
// Render to the stencil
m_upStencilMaskSprite->render( matrix );
// Re-enable color and depth
glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );
glDepthMask( GL_TRUE );
// Where a 1 was not rendered
glStencilFunc( GL_EQUAL, 0x1, 0x1 );
// Keep the pixel
glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
for( int i = m_visStartPos; i < m_visEndPos; ++i )
m_pScrollControlVec[i]->render( matrix );
// Finished using stencil
glDisable( GL_STENCIL_TEST );
Vulkan has the same stencil reference state, as well as stencil function state.
I assume you mean the VkStencilOpState. I understand that but the problem is I assume I need to write something to the stencil buffer. I just don’t know how that is done and I don’t see that happening in the example I sited.
Working with the Vulkan stencil buffer is very different then what I did with OpenGL or DirectX. I need a simple, stencil buffer example because I’m just not getting it.
No, it isn’t. Vulkan stencil operations work in exactly the same way as it does in OpenGL or D3D. If you can understand what that OpenGL code you posted is doing, then you understand exactly how it works in Vulkan too. The only difference is where you put the stuff. All you need to do is map those things to stuff in Vulkan.
You say “I need to write something to the stencil buffer.” Is that not exactly what “m_upStencilMaskSprite->render( matrix );” does? The state setting before that rendering call in OpenGL becomes state that is part of the pipeline object in Vulkan. The state set by the glStencil* and glEnable(GL_STENCIL_TEST) are in VkStencilOpState. The depth write mask is found in VkPipelineDepthStencilStateCreateInfo. And the color write masks are found in the array of VkPipelineColorBlendAttachmentState in VkPipelineColorBlendStateCreateInfo.
There is pretty much a 1:1 mapping between OpenGL function calls and Vulkan pipeline structure state when it comes to stencil stuff. Yes, you have to have a separate pipeline object for the mask render compared to the render using the stencil mask. But that’s it.