Hi all,
I’m getting some weird artefacts at the edges of some objects when I use CSAA with 16 coverage samples and 4 colour samples:
This occurs on a GeForce 8800 GTS and 9800 GTX, both with 175.16 drivers.
Does anyone have any idea what I’m doing wrong that could cause this?
After further investigation, I’ve discovered that using a custom varying in a GLSL shader instead of gl_Color causes this.
e.g. using the following vertex shader:
varying vec4 col;
void main(){
gl_Position = ftransform();
vec3 normal = gl_NormalMatrix * gl_Normal;
gl_FrontColor = col = vec4(normal.zzz, 1.0);
}
this fragment shader works fine:
varying vec4 col;
void main(){
gl_FragColor = gl_Color;
}
but this fragment shader results in artefacts:
varying vec4 col;
void main(){
gl_FragColor = col;
}
Full code available here if you want to see for yourself (requires glew): http://paste.cbwhiz.com/282
This is because gl_FrontColor gets clamped to [0.0, 1.0], and a custom varying does not. 
How does that explain the artefacts? The values stored in col by the vertex shader are already between 0 and 1. Floating point errors may cause the values to fall outside that range slightly, but that shouldn’t result in outputting completely the wrong value.
That looks odd. Have you tried using centroid interpolation? E.g. (in both shaders):
#extension GL_EXT_gpu_shader4 : require
centroid varying vec4 col;
I’d be interested in knowing whether a clamp() helps or not with the user-defined varying… perhaps it’s a derivative issue (in which case centroid may help as Xmas pointed out).
Thanks Xmas - centroid fixed it! So it seems nvidia’s extrapolation method is broken. I’ve submitted a bug report to them so hopefully they’ll get it fixed soon and it won’t be necessary to declare every single varying as centroid!
HexCat - clamping in both the vertex shader and fragment shader didn’t make a difference.
Yes I’m sure, that’s the first article I read. Centroid definitely shouldn’t be necessary when a fragment shader is as simple as the one I posted above.
I look at your code and me too I just think it’s a centroid problem. If you want to prove it’s a NVIDIA problem, I think you should improve your test.
gybe - please elaborate. And if you can tell me how to rewrite the shaders so that centroid isn’t necessary then please do!
I think I’ve figured it out now - for triangles at a near-perpendicular angle to the viewport, the w-coordinate may be extrapolated to a negative number, which combined with a negative extrapolated varying result will giving large values.
If this is correct, if you’re using multisampling and your geometry may be viewed from any angle then always declare your varyings as centroid (with the exception of using dFdx and dFdy) or you may get artefacts.
This needs to change its recommendations accordingly.