Sampling enviromental map in deferred shading

I am using deferred shading, the gbuffer is composed of normals, metallicness ,roughness, and the depth. However inside the final pass (lighting calculations) I’ve noticed that the background of my final image is not being cleared, It’s actually participating of the light calculation. While trying to sample irradiance and enviromental map, the background is actually sampling it too, using the clear value of my normal (0,0,0,0,0) of the normal attachment. So the result is really weird. I’m using loadOp_Store in every attachment.

GBuffer Pipeline

std::vector<VK_Objects::ATRIBUTES> atributes = 
 
{VK_Objects::ATRIBUTES::VEC3,VK_Objects::ATRIBUTES::VEC3,VK_Objects::ATRIBUTES::VEC2 
};

std::vector<std::vector<VK_Objects::ATRIBUTES>>att{ atributes };

pipelineInfo.atributes = att;
pipelineInfo.colorAttachmentsCount = 3;
pipelineInfo.cullMode = VK_CULL_MODE_NONE;
pipelineInfo.dephTest = 1;
pipelineInfo.depthBias = 0;
pipelineInfo.rdpass = &renderpass->passes["G_BUFFER"]->vk_renderpass ;
pipelineInfo.frontFaceClock = VK_FRONT_FACE_CLOCKWISE;
pipelineInfo.vertexOffsets = { 0 };
pipelineInfo.subpass = 0;

Gbuffer renderpass

depthAttachment.description.samples = VK_SAMPLE_COUNT_1_BIT;
depthAttachment.description.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
depthAttachment.description.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
depthAttachment.description.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
depthAttachment.description.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
depthAttachment.description.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
depthAttachment.description.finalLayout = VK_IMAGE_LAYOUT_DEPTH_STENCIL_READ_ONLY_OPTIMAL;
depthAttachment.description.flags = 0;

depthAttachment.reference.attachment = 3;
depthAttachment.reference.layout = VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;


VK_Objects::RenderAttachment metallicRoughnessAttachment;
metallicRoughnessAttachment.description.format = format;
metallicRoughnessAttachment.description.samples = VK_SAMPLE_COUNT_1_BIT;
metallicRoughnessAttachment.description.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
metallicRoughnessAttachment.description.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
metallicRoughnessAttachment.description.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
metallicRoughnessAttachment.description.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
metallicRoughnessAttachment.description.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
metallicRoughnessAttachment.description.finalLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
metallicRoughnessAttachment.description.flags = 0;

metallicRoughnessAttachment.reference.attachment = 1;
metallicRoughnessAttachment.reference.layout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;

VK_Objects::RenderAttachment NormalsAttachment;
NormalsAttachment.description.format = format;
NormalsAttachment.description.samples = VK_SAMPLE_COUNT_1_BIT;
NormalsAttachment.description.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
NormalsAttachment.description.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
NormalsAttachment.description.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
NormalsAttachment.description.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
NormalsAttachment.description.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
NormalsAttachment.description.finalLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
NormalsAttachment.description.flags = 0;

NormalsAttachment.reference.attachment = 2;
NormalsAttachment.reference.layout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;


VK_Objects::RenderAttachment albedoAttachment;
albedoAttachment.description.format = format;
albedoAttachment.description.samples = VK_SAMPLE_COUNT_1_BIT;
albedoAttachment.description.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
albedoAttachment.description.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
albedoAttachment.description.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
albedoAttachment.description.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
albedoAttachment.description.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
albedoAttachment.description.finalLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
albedoAttachment.description.flags = 0;

albedoAttachment.reference.attachment = 0;
albedoAttachment.reference.layout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;

VK_Objects::Subpass subpass;
subpass.description.resize(1);
subpass.dependencies.resize(2);

I’m not trying to render any kind of cubeMap. I’m just sampling it to use in cook_torrance shading.
What am I missing? any clue ?

I may be missing something, but is your lighting pass shader doing anything to detect whether the gbuffer values it operates on are from the background vs. a scene object? Or alternatively do you initialize/clear your gbuffer textures to values that produce meaningful results when used as input to your lighting shaders?

I fill the values of the attachment on a previous pass. I don’t know what should I do to separete the background from the scene. Well, all the attachments of gbuffer are using LOAD_OP_CLEARon loadOP field.

Well, I mean when you are computing lighting you are presumable rendering some geometry that represents the light source’s area of influence (or a full screen quad) in order to trigger the lighting fragment shader invocations. That fragment shader will sample your gbuffer and compute lighting based on what it finds. Now, as an example if you clear the texture that stores the normals to 0,0,0 you would use an invalid normal as input to your lighting computations.
To avoid that you can make sure that you clear to some value that is valid and makes sense (might be difficult as most times you probably do not want any lighting applied to the background) or do an early test in your lighting fragment shader for an invalid normal. That would tell you that the fragment belongs to the background and should not have lighting applied.

“To avoid that you can make sure that you clear to some value that is valid and makes sense (might be difficult as most times you probably do not want any lighting applied to the background) or do an early test in your lighting fragment shader for an invalid normal. That would tell you that the fragment belongs to the background and should not have lighting applied”

Would you give me an example on how do I do that?

If the normal you read from the gbuffer in your lighting shader is approximately (0,0,0) or similar has a length close to 0 perform a discard or alternatively (assuming you use additive blending to combine the lighting) return black color from the shader.

Ok, I got it now. I was really confused because I had already implemented a deferred renderer before and It was working fine, the reason why is because I was actually using an if statemente inside the lighting shader to only perform lighting in the pixels I wanted to. However, using if statements or discard operations seems like very bad for performance. I wonder if there was a better to deal with this situation.

As I mentioned if you don’t want to make a distinction between background and foreground pixels you’ll have to arrange for your lighting shader:

  • still work with the values you clear your gbuffer to as inputs, e.g. it should not perform a division by the length of the normal if you clear the normal buffer to (0, 0, 0), since that would be a division by zero for a background pixel
  • produce a value that does not add any lighting to the background pixels. Again if you use additive blending to accumulate the contributions from each light source your lighting shader can just return black, i.e. (0, 0, 0) since that does not change anything when adding it to what is already in the render target.

If and how exactly that can be done depends on the lighting math that your shader performs. Some of the math may already work out the way you’d want it, e.g. if you did just some diffuse lighting the dot product between the surface normal ((0,0,0) for a background pixel) and the light direction is conveniently 0 and just makes the whole diffuse term disappear.