Very strange lighting (in tangent space)

OpenGL OpenGL: Basic Coding
1

For context, my renderer first does an ambient pass, and then individiual lighting passes with additive blending.

I don’t even know where to begin, to be honest. Here’s my vertex shader and fragment shader. Both of them are compiled twice, for the ambient and directional light pass which has the DIRECTIONAL_LIGHT macro defined by the rendering engine.

Firstly, notice how I ignore the normal mapping, even though the shaders work in tangent space. This is just to be sure that the normal mapping isn’t the problem. The UVs aren’t the problem either, so this must be some basic mistake I’ve done in the diffuse part.

Here’s a video which showcases what happens: https://streamable.com/matvm. See how the right half of the ball seems to be lit fine, while the left side is weirdly darker (the model sits at identity and the light moves around it).

I also tried not doing the ambient pass, and immediately doing the lighting pass right after clearing the buffers (still with additive blending, but the object color is now completely white), which also produced something strange: https://streamable.com/ezgfc. Stare at the right half of the ball and see four different quarters of the right half be different (transparent black, transparent white, solid black, solid white).

I’ve also tried drawing the values of a_tan directly: https://streamable.com/zonay.

Sorry for the confusing explanations, most of this is just random observations.

2

A couple of points:

  1. GLSL has an inverse() function. There’s nothing wrong with your version, although the built-in may be more efficient. But it would be better to pass in the normal matrix as a separate uniform rather than computing it anew for each vertex.
  2. The tangent vector should be transformed by the model matrix, not the normal matrix. The whole point of using the inverse-transpose is that it preserves dot products (i.e. dot(inverse(transpose(M))*n,M*v)=dot(n,v)) even if M isn’t orthonormal.
  3. If the model matrix consists solely of rotation, uniform scaling, and/or translation (no non-uniform scaling or shear), all of this is moot; you don’t need a separate normal matrix.

Uh, not really. Can you elaborate?

From that, it looks like the UVs are mirrored. There shouldn’t be a visible seam between the quadrants. Also: do you have the components in the correct order? The normal map should be predominantly blue (<128,128,255> = #8080FF), i.e. normals should mostly be aligned with the surface normal.

Well those don’t look right. Are you storing tangents in the model, or calculating them from the UVs?

3

I’m targeting GLSL 120, which doesn’t have it.

Could you explain further? Should I replace line 42 with vec3 T = normalize(vec3(u_mmat * vec4(a_tan, 0)));?

Good to know, will probably go with this then.

Sorry, old text and new commit.

That’d be strange if UV’s were broken, because the assets were originally from a DirectX game. For the normal maps I used ImageMagick to invert the green channel.

They’re exported with a Blender script. So far no one who knows the Blender API has confirmed that my exporter is at fault.

4

Yes.

Well, the tangents appear to be mostly continuous except for the poles (where you’d expect there to be issues if you’re just mapping U to longitude and V to latitude) and that one weird bit with the 3 triangles which don’t match each other or their surroundings.

So the discontinuities at the quadrant boundaries suggest that the UVs are discontinuous (which isn’t necessarily a problem, but you’d expect the tangents to also be discontinuous at those points). But even that wouldn’t have much of an effect if the normal map was mostly flat, which is why I suspect that the components may be mis-ordered. The blue channel should be the normal, with red and green for T and B. Swapping T and B wouldn’t have much effect in areas where the normal map is flat, but using either T or B for the normal would. Are you loading a BGR texture as RGB?

6

It’s loaded with the same function as every other texture, and the normal map seems to be mostly light blue: https://streamable.com/cblyl.

But I tried drawing the UV data directly, and yeah, they are discontinous: https://streamable.com/ke4xb, but I don’t know why the tangents aren’t, then. So is it an exporter issue in that case?

7

Discontinuous UVs aren’t automatically a problem, so long as the orientation doesn’t change across seams (or if the tangents change accordingly).

What happens if you just ignore the normal map and set vec3 nrm = vec3(0,0,1) in the fragment shader?

8

There are still seams between the octants, and some octants still have strange lighting.

9

I mean… it has to be something in the diffuse part probably, the artifacts are still there even when I don’t read any textures and draw just the diffuse value (Same artifacts as this video I’ve already showed in OP, where some quadrants have lower diffuse values for no apparent reason).

10

Have you checked that the normals (a_nrm) are correct? For that model, they should be approximately equal to normalise(a_pos).

11

They seem to be fine, at first it was green on the top.

12

I can’t really tell from that. Can you try diffuse lighting but using a_pos in place of a_nrm?

13

Seams still there:

14

I am now 100% sure that it’s the TBN matrix’s fault. If I remove any mention of it from the shaders and instead pass all the positions in world-space the lighting starts working properly.

Another thing I’ve observed is that drawing normalize(vec3(u_mmat * vec4(a_tan, 0))) or with a_nrm produces a black color, while directly drawing a_nrm or a_tan works fine (tested with a plane model).