Does anyone know if there is an efficient way to mark fragments based on a calculated value with different stencil values in OpenGL? Or other recommended way nowadays to do a pre-pass that is capable to do a conditional tagging on the rendered scene?
The only approach I see so far, is to render the scene N times and use the discard keyword for fragments that fail the test. But that seems to be very expensive.
I’m not sure exactly what you’re trying to do. If you need to write a stencil value from the fragment shader, ARB_shader_stencil_export lets you do that if your hardware supports it.
But I’m not sure what this has to do with “conditional tagging on the rendered scene?”
I’m trying to find a way how to quickly group fragments into batches for further processing in deferred rendering.
If the scene has 3 types of fragments, each of them with a complex shader, I would prefer to do a pre-pass with a simple shader which which would assign the fragment to one of the bins. Then, it would be fast to just bind shader A, stencil A, fragments A, render, then switch to next batch and so on.
How can you use multiple complex shaders in deferred rendering otherwise?
If you don’t have ARB_shader_stencil_export, you can still write a specific value to the stencil buffer in ceil(log2(N)) passes, if you write one bit of the stencil per pass. However, I’d just use a shader that implements all three types with dynamic branching. Branching may be slow, but using several passes and a whole buffer just to store an intermediate result is definitely worse.