How to emulate coverage to alpha in vulkan

Vulkan
1

Hello,

I am porting an existing OpenGL 1.3 2D renderer to vulkan. This renderer uses GL_POLYGON_SMOOTH and GL_LINE_SMOOTH extensively.

I am looking for a way to compute the (approximate) pixel coverage of a fragment.
My current approach involves rendering into a multisampled image with VK_SAMPLE_COUNT_16_BIT, and using gl_SampleMaskIn in the fragment shader to compute the final alpha value:

float alpha = popcount(gl_SampleMaskIn[0]) * (1.0 / 16.0);
gl_SampleMask[0] = 0xFFFFFFFF;

This seems to work ok, but it’s very wasteful - i don’t actually need a multisampled image, I just need the rasterizer to compute the sample mask for me.

Things i already looked into:

  • VK_EXT_multisampled_render_to_single_sampled - not very widely supported.
  • Using gl_SampleMask[0] = 1 and VK_RESOLVE_MODE_SAMPLE_ZERO_BIT. This requires integer formats, and there’s no integer format that supports color_attachment (at least on my setup).
  • Gemini claimed that one could set rasterizationSamples = VK_SAMPLE_COUNT_16_BIT in the pipeline creation and then render into a non-multisampled image anyway, but validation layers disagree (and i got device_lost anyway when i tried it).
  • Some discussion in “Functional equivalent to GL_LINE_SMOOTH?” on this forum, but doesn’t cover POLYGON_SMOOTH

I know that multisampling is strictly better than coverage-to-alpha, but i am looking to match the existing opengl implementation as much as I can.