MSAA - Detecting edge fragments

[sqrt_1]

I would like to insert some special GLSL code that handles the edge fragments differently when using multisample anti-aliasing.

I am testing some techniques where I am stretching a lighting texture over the screen and I think it would be possible to use the texture lookup of the nearest polygon edge for AA fragments. (Working on the assumption that the lighting conditions on the fragment are similar enough to the nearest polygon edge)

Basically I would like to do this in a shader:

if(edgeSample)
{
// Find nearest polygon edge

// Offset texture lookup to go-to nearest edge
}

I am thinking that perhaps centroid sampling may give me this information (somehow) but I was wondering if anyone had any other ideas?

[namespace]

This is just an idea:

Sample your Z-Buffer with a pattern like box or triangle and
transform the z values back to world- or eyespace and calculate the difference
between the values.
A sample would be considered an edgesample if the delta of the z values exceed
a specific epsilon.

[oc2k1]

It is possible to use the difference of a “varying” and “centroid varying” if that is != 0.0 a fragment is on a edge.

[Humus]

That would only detect cases where the fragment fell outside the polygon, but not when the pixel center is covered yet some samples in the pixel are outside the primitive.

[Xmas]

@sqrt[-1

]// Offset texture lookup to go-to nearest edge

I don’t exactly understand what you want to achieve, but this sounds almost like you’d just want to use centroid texture coordinates directly.

[sqrt_1]

@Humus

That would only detect cases where the fragment fell outside the polygon, but not when the pixel center is covered yet some samples in the pixel are outside the primitive.

In my case this would be OK.
What are the rules for multisampling? - Is it the case that when the pixel center is covered yet some samples in the pixel are outside the primitive, that even with centroid sampling the values are at the pixel center?

I was also hoping I could assign a value in the vertex shader to be interpolated - then if I detected a value out side the valid range - I knew I had fallen off the polygon.

Eg. Kinda like using the example in the pipeline
http://www.opengl.org/pipeline/article/vol003_6/
and detecting when the value was less than zero.

I am unsure what values I could assign per-vertex to detect this however.

I will try the centroid diff idea and see what happens.

[sqrt_1]

@Xmas

[quote=sqrt[-1]]// Offset texture lookup to go-to nearest edge

I don’t exactly understand what you want to achieve, but this sounds almost like you’d just want to use centroid texture coordinates directly. [/QUOTE]

Yes, that would get me inside the polygon boundary, however I want to sample from a fullsceen texture without going outside the polygon with point sampling.

I suppose I could take the centroid values then round them down/up to the nearest pixel center. (not sure if/how it would work)

[Xmas]

@sqrt[-1

What are the rules for multisampling? - Is it the case that when the pixel center is covered yet some samples in the pixel are outside the primitive, that even with centroid sampling the values are at the pixel center?

The rules AFAIK:

• without centroid, varyings can be evaluated at any point inside a pixel
• with centroid, varyings can be evaluated at any point inside the convex hull of covered samples.

@sqrt[-1

Yes, that would get me inside the polygon boundary, however I want to sample from a fullsceen texture without going outside the polygon with point sampling.

What is a “fullscreen texture”? Do you mean you want a 1:1 mapping of texels to pixels? What geometry are you rendering?

I still don’t see what you’re actually trying to achieve.

[Humus]

@Xmas

The rules AFAIK:

• without centroid, varyings can be evaluated at any point inside a pixel
• with centroid, varyings can be evaluated at any point inside the convex hull of covered samples.

Without centroid, varyings are evaluated at the pixel center. With centroid, it could be any point in the pixel that’s covered by the primitive. If the pixel center is covered, it makes the most sense to use that, but I’m not entirely sure if that’s required.

[Humus]

@sqrt[-1

]I was also hoping I could assign a value in the vertex shader to be interpolated - then if I detected a value out side the valid range - I knew I had fallen off the polygon.

Eg. Kinda like using the example in the pipeline
http://www.opengl.org/pipeline/article/vol003_6/
and detecting when the value was less than zero.

I am unsure what values I could assign per-vertex to detect this however.

Well, with centroid sampling you never fall outside the polygon. That should solve it for most problems with artifacts. However, if I understand your particular problem right, I suppose this wouldn’t be enough. One idea might be to interpolate barycentric coordinates, so output (1,0,0), (0,1,0) and (0,0,1) to a texture coordinate for the vertices in a triangle. Then check if any of the coordinates is less than zero. If so, you’re outside the primitive. I made quick test with this and from the looks of it appears to work, got some white pixels along the triangle edge. Then with centroid sampling enabled they disappeared.

[Xmas]

@Humus

Without centroid, varyings are evaluated at the pixel center.

That’s the most likely reality, but there’s no such guarantee for multisampled rasterization. See section 3.5.6 of the spec (Polygon Multisample Rasterization). That’s a change from GL_ARB_multisample, by the way, mainly to allow supersampling.

With centroid, it could be any point in the pixel that’s covered by the primitive. If the pixel center is covered, it makes the most sense to use that, but I’m not entirely sure if that’s required.

It’s not required, but again highly likely.

[sqrt_1]

Well, with centroid sampling you never fall outside the polygon. That should solve it for most problems with artifacts. However, if I understand your particular problem right, I suppose this wouldn’t be enough. One idea might be to interpolate barycentric coordinates, so output (1,0,0), (0,1,0) and (0,0,1) to a texture coordinate for the vertices in a triangle. Then check if any of the coordinates is less than zero. If so, you’re outside the primitive. I made quick test with this and from the looks of it appears to work, got some white pixels along the triangle edge. Then with centroid sampling enabled they disappeared.

Yeah, that was what I was thinking of doing, but I can’t figure out how you could generate such coordinates in a vertex shader. (Could probably in a geometry shader however)

[Humus]

Well, for non-indexed primitives you can just supply another vertex attribute with the coordinates. For indexed I suppose the geometry shader is the only option (or simply convert it to non-indexed).

[sqrt_1]

Thanks everyone for all the advice and help, I’ll let you know how it turns out.

(Oh and Happy Birthday Humus )

[Humus]

Thanks! Only 37 years left to retirement, woot!