Sounds like what you want is to “disable” SAMPLE_ALPHA_TO_COVERAGE and instead enable BLEND for your translucent objects.
What SAMPLE_ALPHA_TO_COVERAGE does is convert the fragment alpha into a multisample dither mask, and renders a subset of the samples opaquely. Then when the MSAA downsample happens you get a very quantized alpha transparency look. This is also sometimes called screen-door transparency.
You just saved my day! I’m using GL_SAMPLE_ALPHA_TO_COVERAGE for so long now… Just recently I wondered why my downsampled alpha-values are so small and figured, that multisampling and a-2-c together basically result in “squared alpha”.
SAMPLE_ALPHA_TO_ONE fixes this!
Really? I’m not sure but that doesn’t sound quite right.
I’m guessing you had multisampling, SAMPLE_ALPHA_TO_COVERAGE, and BLEND all enabled together. Is this right?
If so, then I’d definitely believe you should see a “squared alpha” type effect (only some samples would be blended based on alpha, and additionally those would be dimmed by alpha). I’d also believe that, as you did, setting SAMPLE_ALPHA_TO_ONE should probably fix the visual artifact, but you’ve still got a needless BLEND hit there.
Alternatively leaving SAMPLE_ALPHA_TO_ONE disabled and just disabling BLEND while using SAMPLE_ALPHA_TO_COVERAGE might also do what you need and be faster.
…unless I’m missing something. …which is entirely possible.
I blend the downsampled image again with another image. The framebuffer alpha channel serves as opacity for this blend operation.
Now image, you’re using alpha=0.25 when you initially render the 4x multisampled image. This alpha value is used as coverage mask, so 1 of 4 of the samples gets alpha=0.25.
When you downsample this image, the resulting pixels gets an alpha value of (0+0+0+0.25)/4 = 1/16. This is not the intended value. Instead it should have written (0+0+0+1.0)/4=0.25.
So I wondered how the hell I’m getting a 1.0 into the fragment alpha channel without losing alpha-to-coverage. The answer is GL_SAMPLE_ALPHA_TO_ONE
(The initial multisampled image is not using blending. Alpha-to-coverage is just used for cheap order-independent translucency)