Overlapping edges of a rectangle.

I want to draw an outlined rectangle. However, the edges of the rectangle overlap when I use a line width greater than 1. This is disturbing when they are translucent, since it leads to highlighted edges.
In addition, the edges are stepped. But I think that can be solved by drawing a point at the edges with an appropriate point size. Nevertheless, these points will overlap with the lines, too.

Any suggestions, ideas what to do?

I can think of using the stencil buffer for example, but unfortunately I already use it for a different purpose (alpha masking). Moreover, I’d have to clear the stencil buffer every time I draw a rectangle, because
I want to draw several overlapping translucent rectangles and these should indeed create highlights at the overlapping parts.


I tested it with GL_LINES, GL_LINE_LOOP and GL_QUADS (with glPolygonMode(GL_FRONT_AND_BACK, GL_LINE)), but always the highlights appear.

I really need some help with it.



This could be a tough one if you can’t use stencil. One option might be drawing the lines first into the depth buffer with color writes disabled, and then draw them again with glDepthFunc(GL_EQUAL). I’m not sure it it looks perfect either, though, but it might be worth a shot. If you want several boxes to overlap, you must draw them back to front.

Also, you might me able to use stencil too. Your alpha mask only requires one bit, and practically all cards support at least 8-bit stencil. So you could draw the mask into the highest bit and use the rest for the rectangles. Stencil functions support bit masking so it’s easy to do. Then draw a rectangle to setting stencil to 1, and failing the stencil test when the stencil already is 1. For the next rectangle do the same, but use 2 as your index instead of 1. And so on. Use bit masks to not let this affect your alpha mask.


Well, your proposed stencil method does not wolk for me because I need all 8 bit of the stencil buffer. Furthermore, this would restrict the number of rectangles I can draw to around 255, right? I want to draw a potentially infinite number of rectangles.

I am not sure if I understood your depth buffer method. When I first draw the rectangle into the depth buffer only and then again with GL_EQUAL… this will draw the whole rectangle including the overlapping edges. I tested it. Maybe I misunderstood something.

I tried glDepthFunc(GL_NOTEQUAL), and that works for one rectangle. I could clear the depth buffer everytime (which is too expensive, I think), or redraw the rectangle with glDepthFunc(GL_ALWAYS) to reset the depth values.
Another variant may be to increase the depth value everytime I draw a primitive (I also draw lines etc). This would restrict me to a finite number of primitves, but much more than the 255 with the stencil method.

Both methods have advantages and disadvantages.

Can you think of a different approach?


I’d still try it with stencil. Clearing stencil every 256 rectangles might not be so bad. Or then you could draw each rectangle twice, second time to erase it from the stencil. Is there some reason why you can’t put your alpha mask into destination alpha where it belongs? You can disable alpha writes by glColorMask if you’re afraid that your translucent rectangles will mess it up.

Another alternative is to draw the rectangles from polygons, not lines. Then you could build them so that they don’t overlap.


You are right, the stencil method is my favourite, too.
I would really like to put my alpha mask to destination alpha, but I don’t see how that would be possible. I currently use a glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA). So, I want to have the typical effect of translucency. That gives me a blending equation like this:

(RsAs + Rd(1-As), GsAs + Gd(1-As), …, …)

with (Rs, Gs, Bs, As) the source color, (Rd, Gd, Bd, Ad) the destination color. I think with an alpha mask in the destination alpha, I want to get an equation like:

([RsAs + Rd(1-As)]Ad, [GsAs + Gd*(1-As)]*Ad, …, …)

right? But how?

Using glColorMask() is a good idea.

I thought about doing it with pologons, but how do I clear the interior of the polygon, without affecting what is already drawn there?


So the second equation is what you want? Easy:

  1. Draw mask in destination alpha
  2. Draw your stuff with the normal blend function, SRC_ALPHA, ONE_MINUS_SRC_ALPHA. Use colorMask to protect your alpha mask
  3. Draw a fullscreen quad with blending func GL_ZERO, GL_DEST_ALPHA.

I think that’s it. I didn’t understand the part about clearing interior of a polygon.


Gah, damn, you are right.
I’ll try it that way.

Thank you very much.

The method of JustHanging would not create the same effect as with the stencil buffer, right?

The above blending equation will be fullfilled, but drawing a quad over the whole scene will affect every pixel. I mean, if ingo draws a rectangle with the first glBlendFunc and then the fullscreen quad to apply the alpha maks, then every other pixel drawn before the rectangle will be affected, too. So, if he draws several rectangles (and fullscreen quads), the alpha mask will be applied to some rectangles several times. But I think this effect is not desired. At least not by me.

Is there a possibility to avoid this without the stencil buffer?

No, you draw only one fullscreen quad, after all the rectangles have been drawn. It should work, altough it’s hard to tell, since I don’t know what exactly you’re trying to do.


Would you use glPolygonOffset(…) in this case?

Hmm, I’ll try to explain what I want to do.

I don’t want to apply the alpha mask after having drawn the scene but everytime I draw something. Not to the whole scene, but only to the things I draw at the moment. This would work with the stencil buffer, wouldn’t it? The stencil function is applied everytime something is drawn, right?

So I guess your mask is changing somehow while you draw your scene, otherwise I don’t see the problem. One option is to store the mask in a texture and project in on the geometry.


Yes, my mask may change but this is not the only problem. Applying the mask after the scene is drawn is a different effect than applying the mask in every step as it would be with the stencil buffer.

I’ll try to explain. Let’s do it for a single pixel/fragment. When I draw three translucent pixels on top of each other and then apply the mask is simply different from drawing one pixel with the mask applied then drawing the second pixel with the mask applied and so on.

You can check it with the blending equation in the style ingo did it. The alpha mask coefficient will be at different places. When you apply the alpha mask immediately you get

[CsAs + Cd(1-As)]*Ad

for one pixel (Cs/Cd: source/destination color component, As/Ad source/dest. alpha component),

{CsAs + [CsAs + Cd*(1-As)]Ad(1-As)}*Ad

for two.

When you apply the alpha mask after having drawn the scene you get

{CsAs + [CsAs + Cd*(1-As)]*(1-As)}*Ad

for two pixels.

(I hope I did not make any mistakes)

I hope I did not make it too complicated.

But could you explain your method with the texture and applying it to the scene? I am still learning OpenGl


Ok, I get it. I was thinking about a binocular kind of view where the applying the mask to the entire image would suffice.

So the texture thing then. Put the mask in the alpha channel of a white texture. You can either precompute it or use glCopyTexImage or render-to-texture to update it on the fly. Bind this texture to all objects using GL_MODULATE texture enviroment. Use multitexture if you have other textures too.

Use automatic texture coordinate generation to project the texture on the geometry. That is, load rows 1, 2 and 4 of your projection matrix to GL_EYE_PLANE of s, t and q coordinates using glTexGen (or was it glTextureGen, whatever). Notice that when you get the projection matrix with glGetFloatv, it will be in column-major order, so you’ll have to transpose it before loading it to texgen. You will also have to apply a scale and bias to it, since texture coordinates vary from 0 to 1 instead of -1 to 1. That’s easiest to do with texture matrix, like this:

glTranslatef(0.5, 0.5, 0);
glScalef(0.5, 0.5, 0.5);

You could also apply these transformations into the projection matrix before you load it to texgen and avoid using the texture matrix, but that might be a bit more complicated.


OK, I’ll try it (although it seems to be a bit complicated). Thank you.