Please Help! Sorting by depth for blending modes...

For good looking transparency effects, I know that I should enable depth buffering, draw all my solid polygons, and then draw my transparent polygons sorted from back to front.

My question is how do I sort them? Can I use an OpenGL function to calculate the distance from the eye point (0,0,0) to a translated/rotated point on my polygon in 3-space? Or do I need to calculate the distance myself? How? I know the formula for distance, but what’s the best way to account for the rotation and translation? I’ve looked in the OpenGL Bible and can’t find the answer.

What point is best to use for sorting? The nearest, farthest, or mid-point? Why?

If you know of a good tutorial for this, please post a link. I’ve been to NeHe’s but the actual sorting process was not described in enough detail for me to follow.

Thanks in advance for the help!

Ok, to get the depth, I have my graphics engine do all the transformations, i.e. I do not use glRotate or glTranslate. And I use the mid-point of each triangle for sorting. This works most of the time just fine. However if you have a transparent polygon intersecting another, to get the correct rendering you would need to split the polygons along the line they intersect and then sort the resulting polygons. Avoid intersecting transparent polygons and you don’t have worry about splitting.

and since for sorting you’re using the distance only for comparison, i think you can get away with a few shortcuts in the distance
formula.

for instance:
if
(sqrt( (object1.x - camera.x)^2 + (object1.y - camera.y)^2 ) < sqrt( (object2.x - camera.x)^2 + (object2.y - camera.y)^2 )

then this implies that

((object1.x - camera.x)^2 + (object1.y - camera.y)^2) < ((object2.x - camera.x)^2 + (object2.y - camera.y)^2)

/*
[note: this step won’t work, see relic, below]

which implies that

(abs(object1.x - camera.x) + abs(object1.y - camera.y)) < (abs(object2.x - camera.x) + abs(object2.y - camera.y))
*/

basically you can avoid all the calls to sqrt() and probably avoid multiplication as well.

[This message has been edited by phlake (edited 07-12-2000).]

"For good looking transparency effects, I know that I should enable depth buffering, draw all my solid polygons, and then draw my transparent polygons sorted from back to front. "

The trick is to enable depth testing but disable depth writing (glDepthMask(GL_FALSE))for the transparent polygons. Then, depending on the blending function, you can get away with intersecting transparent faces without splitting or sorting at all.

If aa <= bb then abs(a) <= abs(b) but be careful with the sum of values less than 1.0.
Example:
0.5 * 0.5 + 0.5 * 0.5 = 0.25 + 0.25 = 0.5
0.1 * 0.1 + 0.9 * 0.9 = 0.01 + 0.81 = 0.82
but 0.5 + 0.5 == 0.1 + 0.9 !!

And the actual implementation of that, because abs() resp. fabs() is a function call and ‘*’ not.
You could generate an absolute value by clearing the highest bit of the IEEE float value or calculate it with an if-statement, but since a multiple takes 3 clocks or less on a PentiumPro architecture it might not be worth thinking about it.

[This message has been edited by Relic (edited 07-12-2000).]

Be careful Relic, the sorting of transparent polygons is not dependent on just the blending mode of the polygon you are wanting to draw at any given moment, but also on the blending mode of other polygons that may lie completely behind or in front. For example consider one polygon using glBendFunc(GL_SRC_ALPHA,GL_ONE) and another polygon that lies above or below that uses glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA). The resulting color is dependent on the order in which they are drawn, meaning even glBendFunc(GL_SRC_ALPHA,GL_ONE) blended polygons would need to be sorted with the glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA) blended polygons.

[This message has been edited by DFrey (edited 07-12-2000).]

i thought that last step seemed screwy. and you’re right about multiplication, relic, it’s probably not worth avoiding. not to mention that checking aa < bb looks more natural, and would result in cleaner code.

abs() and fabs() make great tertiary macros, though. a = a < 0 ? -a : a;

the incorrect results worry me more.

[This message has been edited by phlake (edited 07-12-2000).]

Yeah, forget about that “less than 1.0” thing. I haven’t thought about that long enough. 0.0 < a < 1.0 is just the range where a^2 < abs(a).
The formula is also false for other values.

I agree, sometimes you’ll have to sort.
As I said, depending on the blending mode.

What just comes to my mind is, if there exists any blending mode combination which emulates real world translucency, where for example red and green filters together block all light and an opaque (black) area is left?
Hmmm… I’ll have to think about it.

Yes, such a subtractive blending mode does exist as an extension in some OpenGL implementations.

Just enable depth masking to draw your solids, then dissable it to draw your transparent objects, then re-enable it just before you swap buffers…This is rather crude, but worked great for my simple fly-through program.

-Noah Desch