confused about handedness/winding in tangent space

OpenGL
1

I have a basic tangent space normal mapping shader going. It works OK for basic diffuse lighting, but is having serious handedness problems for specular lighting. This is using objects exported from Blender, and it seems Blender’s triangle-ization step on obj export switches handedness a lot. In fact, if you bake a normal map from blender, you can see that it compensates for this in the normal map. But I’m hand painting my normal map, plus I need to get tangent space specular lighting working.

I’m getting myself really confused about all this :-(. I’m using the technique from here, the same math but rewritten for my own environment: http://www.terathon.com/code/tangent.html

I calculate both tangent and bitangent in C++ code like that, then send them as user attributes to the VS.

One problem I think exists: these lines “tan1[i1] += sdir;” which calculate the tangent and bitangent are going to add in sdir/tdir from opposite windings, aren’t they? It seems like that won’t work in such a case, unless I’m missing something.

On that theory, I’m trying to calculate the handedness in the C++ code that calculate tangent/bitangent, and add in the right sense of sdir/tdir. But it isn’t working right; my specular lighting is still bad across some of the triangle boundaries (but works across others).

Am I even right about the problem here? And if I am, is it possible to fix this right there in CalculateTangentArray?

My hacked up attempt to do this is like this, right before sdir/tdir get added to the two tangent vectors which are accumulating for each vertex shared by several faces:

bool rightHand = (sdir.cross(tdir)).dot(norms[tri]) >= 0.0;
if (!rightHand) {
tdir *= -1.0;
sdir *= -1.0;
}

But that’s clearly wrong. I’ve also tried only reversing sdir, and that’s wrong too :). But I’m not sure why.

Any help would be really awesome! Thanks a ton.

Oh, I also have txcoords that wrap 4 times around my object. Is that going to cause any problems for this technique, such as when the txcoord wraps from 1 back to 0?

2

A quick follow-on q: I see the code on that page does calculate a handedness, but it does so after the accumulation into tan1/tan2. I don’t understand that: isn’t it too late? Seems like triangles from two different handedness could share a vertex, in which case the accumulation doesn’t make sense.

I guess what I don’t understand is: if the handedness is just a function of N & T, since I’m also calculating the bitangent in that C++ code, is there any mathematical reason I can’t just reverse the bitangent right there, instead of passing it to the VS? Most examples I see calculate the bitangent in the VS through a cross product, but my tangent/bitangent might not be orthogonal, so I pass both of them down.

3

I think that triangles with different tangential space handiness should not share a vertex. As you are exporting from Blender, you probably duplicate vertex positions for the triangles that have different normals. Just add the handiness bit as a differentiation factor - and you will not accumulate tangents from opposite handinesses any more.

4

Thanks, Dmitry! I think my problem is something like that. Unfortunately it seems Blender reverses the handedness almost every triangle, so if I split them out every time it changes, it will remove the entire effect of using indexed rendering.

Anyway, I will keep trying… I have some of it fixed now, but there is still a problem where my txcoords wrap on the object.

5

Hm… I’m constructing the tangential space from the UV layer. So On my side the only triangles with reversed handness are those which have UV flipped. There is usually very few seams in a model, so the duplicated vertices are not a big deal.

How do you get tangential space from blender?

6

I’d advise to keep in mind what right handedness means: essentially:

dot(normal, cross(tangent, biTangent) ) usually.

which is what you have, but what happens if you negate both tangent and biTangent? Nothing to the handedness, since cross(A,B)=cross(-A,-B).

Simplest solution: just negate one of tangent or biTangent, that will flip the handedness, or exchange tangent with biTangent (I advise against both!)

Better solution as DmitryM says, if two triangles have different handiness, then duplicate the vertices.