Hello,
I have tried to debug my issue for a couple days, and this is my final attempt at solving the problem. So, here goes.
I have a render pass. The render pass has three subpasses. Two of the subpasses share input assembly bindings and attributes, while the third subpass has different bindings and attributes because it has extra per vertex information for normal mapping.
I could go into more detail about the setup but I have some RenderDoc images that will hopefully make this easier to explain. Here is a frame where there are draw commands in subpass 1 & 2 (the two subpasses that share input assembly binding/attributes). These are a line subpass (1), and a triangle list subpass (2).
https://drive.google.com/file/d/1h-fA7bBepPtn8St12DlOkcHCQZKiz5y_/view?usp=sharing
As you can see there is a nice little doughnut and some axes. (Note: This image shows the state of the color attachment after the draw command for the torus. You can see the draw command in the a list of all the draw commands highlighted in blue in RenderDoc on the left side of the image). Here is another image of the framebuffer at the end of the render pass. You can see that ultimately some UI elements are draw later in the same subpass.
https://drive.google.com/file/d/15WVzvPwdtLDHVLpv1i2DRaR0qPq5AKrU/view?usp=sharing
To recap, these are captures of the color attachments and framebuffer when only two of the three subpasses are issuing commands. So here is the problem: when I add a draw commands to the render pass for the subpass that has different assembly info the torus gets distorted in a strange fashion. The images above were both taken when subpass 1 & 2 where issuing commands, which is hopefully clear when you look at the command list in RenderDoc on the left side of image.
Here is an image from the render pass when subpass 0 (the odd one) has a draw command:
https://drive.google.com/file/d/1nK-6zRRbQafTCtLmRmKfkFFJvCr7PBSc/view?usp=sharing
To me, and I am fairly new to this kind of programming, this type of distortion usually comes from a bad transform or some uniform has not been aligned properly when copied to the device or something. This might be the case but, I am leaning toward it being something I am unfamiliar with, and maybe it will be obvious to someone else with more experience than me.
So here is what makes it strange to me - If I order the subpasses in a different order so that 1 & 2 come before the 0 then the issue goes away and everything draws properly. Here is another example taken from the same RenderDoc capture as the last image:
https://drive.google.com/file/d/1QJhLXCtm5YCyZ3YKvxkfno9smdhJI_fz/view?usp=sharing
This is an image taken directly from the color attachment after the torus draw call (you can verify this in the command list on the left side of the image when compared with the first image). And one more image taken directly from the subsequent command:
https://drive.google.com/file/d/1w2cgorVz77eBgKM8gbotAOdngZ0BLx65/view?usp=sharing
You can see that the torus draws fine at first and then something later in the subpass is distorting vertex values, or discarding fragments or something.
So, here is my question: what could possibly be causing this issue? I know that its nigh impossible for someone looking at these images to know, but I have run out of leads. I have tried to write subpass dependencies that prevent writing to the color attachment until all of the subpasses have done fragment tests. I have really tried all kinds of dependency masking and it appears that I can put any values in and it does not change the outcome. Has anyone else ever had an issue like this and created the magical subpass dependency that fixes the problem? Does this even look like its a fragment discarding problem? I really have no clue at this point.
I should mention that I have the LunarG validation layers turned on and nothing is being reported on that front either. At some point I can just reorder the subpasses and move on, but I would like to try and understand this before moving on since I am just trying to learn this stuff for fun.
Thanks