I’m trying to detect pixels that are located on multisampled polygon edges. Therefore I’m trying to produce a centroid-sampled version of gl_FragCoord. If the centroid-sampled pixel coordinate does not lie on the pixel center (0.5, 0.5), it must be near an edge.
But I’m having troubles generating the pixel coordinate as needed…

noperspective centroid varying vec3 v_pixelpos;
//do perspective divide, transform from NDC to 0..1 range
vec2 pixelpos = ((v_pixelpos.xy/v_pixelpos.z) + 1.0)*0.5;
//multiply by viewport size
pixelpos = pixelpos*vec2(1280.0, 768.0);
//debug output. Most pixels should produce some yellow (0.5, 0.5)
//the edge pixels should produce something different
gl_FragColor=vec4(fract(pixelpos), 0.0, 1.0);

Unfortunately, I only get garbage… what’s wrong with the approach?

@Ilian:
Doing the full transform in the vertex shader and using ‘noperspective centroid’ does not work. Large polygons that need to be clipped by the near plane go completely crazy. I think basically, because out-of-screen polygons produce ‘impossible’ screen coordinates.

That’s why I’m doing the transform in the fragment shader. But I guess, I can shuffle much more of the calculation into the vertex shader, except the perspective divide.

I have now found a way to get it working: Basically, since I’m doing the perspective divide in the fragment shader, I cannot use ‘noperspective’. After I removed that, it started to work.

centroid varying vec3 v_pixelpos;
...
//perspective divide, NDC to 0..1
vec2 pixelpos = (vi_pixelpos.xy/vi_pixelpos.z)*0.5 + 0.5;
//viewport size
pixelpos = pixelpos*vec2(1280.0, 1024.0);
//distance from pixel center
pixelpos = fract(pixelpos)-0.5;
float d=dot(pixelpos, pixelpos);
//pink colour when sampling outside the polygon
if (d>0.01)
gl_FragData[1]=vec4(d, 0.0, d, 1.0);

It works quite well now. But really large polygons that clip the near plane still produce artifacts. I guess due to polygon clipping and inexact interpolation, the computed pixel coordinates start to differ too much from the real pixel coordinate.

Your method indeed works quite well! None of the artifacts I was seeing occur.

Maybe this is because I was insisting on generating pixel-perfect coordinates, which suffered from interpolation and clipping precision issues.

With your method, I just compare two values which get generated and interpolated the same way. Their absolute (interpolated) values have no meaning, though.