GLSL: Two-sided lighting and glCullFace


in the past (using fixed function pipeline) I did the
following to render transparent objects while two sided
sided lighting was enabled to achieve “correct” lighting:

if (o->Transparent)
  // Render backfacing triangles first

  // Now render frontfacing triangles

Now I want to add two sided lighting to my shader.
Therefor I guess I just have to flip the normal if
the current triangle is a backface and then do the
lighting calculations as usual.

Now my questions:

  1. Since I call glCullFace(GL_FRONT) the fragment
    shader should only be executed for backfacing triangles,
    is that correct?

  2. But the vertex shader is called for all faces since
    it works on a per-vertex and not on a per-face basis?

  3. So how to take two-sided lighting into account?
    I guess it should not belong into the vertex shader
    so should I just check my interpolated normal in
    the fragment shader (also used for specular stuff)
    and always flip it for the pass glDrawElements
    pass with glCullFace(GL_FRONT)?

  4. How to handle objects with smoothing groups i.e.
    smoothed normals. I think in this case for the backface
    check I need a “real” normal which is not the normalized
    sum of adjacent triangles?

Help is really appreciated! :slight_smile:

Agree. Front-facing primitives should be discarded before coming to fragment shader.

Vertex shader is executed for all vertices. Primitives are assembled after vertex shader stage.

Not necessarily. Although in the fragment shader you can easily determine side by using built-in variable gl_FrontFacing. In vertex shader the side can be checked by testing the sign of scalar (dot) product of the normal and the light position vector. It is not the same as testing the face orientation according to order in which the verices are defined, but I guess this will serve the purpose.

I’m not sure about this. Can you reformulate the question? It seems that there is no need for what you call “real” normals. In any case, if you are about to implement per pixel lighting, you’ll have different values in each fragment.

Agree. Front-facing primitives should be discarded before coming to fragment shader.

Vertex shader is executed for all vertices. Primitives are assembled after vertex shader stage.[/QUOTE]
Right. Just one thing to add to this to help make it crystal clear. CullFace is for telling the GPU to cull front and/or back-facing tris. Front or back facing in what space? Eye space! Where does the GPU know what the tri coords are in eye-space? After the vertex shader. So, the vertex shader has to run first on the verts, before the GPU can make that facing determination for culling.

It depends. If you’re doing vertex lighting (Gouraud shading), then obviously you’d do your “normal flip” for two-sided lighting before you plugged it into the lighting equation in the vertex shader.

But you said “interpolated normal” which suggests fragment lighting (Phong shading). So you could just interpolate the unmodified normal across the tri, and then flip in the frag shader as needed for two sided lighting.

The question is, could you instead do the flip in the vtx shader if needed, and interpolate the “potentially-flipped” normals across the tri for the frag shader. Sometimes maybe, but you can think of cases with edge-on views of tris where this could go horribly wrong. That is, essentially some tri vtx normals would be flipped and others wouldn’t, leading to a horrible normal mess.

…and always flip [the normal] for the … glDrawElements pass with glCullFace(GL_FRONT)?

That’s one way. Another way (that’s not actually per-spec but that you might consider) is to always flip the normal toward the eye for two-sided lighting. That way you don’t even need to care which “facing” you are processing…

…unless of course you are applying different material properties to the front and the back in the same shader.

Thanks a lot guys! Yeah I’m talking about phong shading.

That was my hidden agenda if this flip could be moved
to vertex shader to save some cycles but I agree with
you that this could mess up things. I ended up using
a uniform float to control the normal flip in the
fragment shader. I set it to -1.0 for the

Also thanks for the hint with gl_FrontFacing. It could
be useful in situations where no glCullFace is used :slight_smile:

Please ignore my last question (4.). I guess I already
answered it myself. I don’t have to care about the normals
generated from 3ds max smoothing groups (i.e. for smooth
shading of spheres). They do not contribute to the backface
detection in the two-sided lighting stuff since I use
glCullFace and not an own backface detection in the shader.

Ugh I just noticed a big mistake in my considerations.
I only have to flip a normal as soon as the dot product
between lightvector and normal is < 0.0. It has nothing
to do with the two backface/frontface passes which are
resulting from glCullFace. So instead of my uniform
float I’ll pass a bool value for two-sided lighting
and then check the result of each dot…

edit: just modified the code and it works fine this way :slight_smile: