I learned somewhat recently about this behavior described in the GL_ARB_multisample spec:
“If MULTISAMPLE_ARB is enabled, and SAMPLE_BUFFERS_ARB is a value of one, then points are rasterized using the following algorithm, regardless of whether point antialiasing (POINT_SMOOTH) is enabled or disabled. Point rasterization produces a fragment for each framebuffer pixel with one or more sample points that intersect the region lying within the circle having diameter equal to the current point width and centered at the point’s (Xw,Yw)”
I’m working in an older fixed-function codebase and this is causing points that are intended to be square to appear circular if GL_MULTISAMPLE is enabled (when rendering to an MSAA back buffer). I’ve always thought of GL_MULTISAMPLE as something you leave on when using MSAA for a final anti-aliased image, and that’s it. Is this something that I should be able to toggle on and off for different pieces of geometry, so I can just get square points in an otherwise FSAAed image? My tests so far indicate that this doesn’t really work. If I draw two points with GL_MULTISAMPLE off, then some other geometry with it on, one of the points appears square and the other round, if I just draw one point it always seems to be round.
We have a path to fall back to textured quads if selectively enabling GL_MULTISAMPLE is not something that is supposed to work, but I wanted to see if anyone knew what the spec says about this first.
I believe that when you disable MSAA with a render buffer that has MSAA, each fragment is drawn in the center of the fragment buffer but the aliasing is still done at the end when the sample buffers are merged. I must admit I would expect a point to be round with MSAA since the whole idea of MSAA is the smooth edges and a point by definition is round. Mixing an old code base with newer features is like to generate slightly different results.
I ran into a similar issue when rendering point cloud data as tiny quads to approximate a surface. I used glEnable(GL_POINT_SPRITE) to force the points to render as quads again, but I’m not sure if this trick will work in the fixed function pipeline.
Yes, you can toggle multisample on and off as you render. However you probably don’t want to do this frequently, as it may produce very small batches. If you’re doing this only once per render for rendering all the points it’ll be fine.
After looking into this I found that toggling multisample off for points and on for other geometry works as expected on Nvidia cards, but on AMD turning multisample on at any time during the frame usually gave all round points or some unpredictable amount of them (I couldn’t figure out what the rule seemed to be).
For example this code shows two round points on my Radeon HD 7800. On an Nvidia card (not sure the model) I have at work it shows two square points. I have the full project available if anyone wants to try it themselves.