No it’s not the “force antialiasing setting”, I am rendering intentionally on a multisampled PBO.
The problem is nVidia has a setting in their drivers which enable gamma correction for antialiasing and, to make the matter worst, it is enabled by default.
With that setting enabled the “antialiasing pixels” have a non-linear blending.
I make an example:
Let’s say that my Blending function is SRC_ALPHA for RGB and ONE for alpha and the framebuffer is black.
I render a white (1.0, 1.0, 1.0) triangle with alpha 0.5.
The expected outcome (and it is) is to have for triangle pixels (0.5, 0.5, 0.5, 0.5).
For antialiasing pixels I’m expecting lesser values, but always with the components all equal, for example (0.3, 0.3, 0.3, 0.3).
When the antialiasing gamma correction is enabled, this is no longer true and some pixels have a wrong value like (0.4, 0.4, 0.4, 0.3).
Since I’m using a post-processing routine on that buffer, this leads to incorrect results, unless the setting is disabled.
The post processing part, is done on the CPU and is simply bringing the buffer to non-premultiplied alpha (dividing the RGB by the alpha value).