There are two options:
- Calculate surface normals in the fragment shader, using
vec3 normal = normalize(cross(dFdx(pos), dFdy(pos)));
pos is the interpolated position in some coordinate system,
normal will be the face normal in the same coordinate system.
- Use the
flat qualifier on the varying (vertex shader output, fragment shader input) for the normal. The value for the last vertex of each triangle will be used for every fragment, rather than interpolating the values from three vertices.
That way, a vertex can be shared by multiple triangles with distinct normals. The vertex holds the normal for one of the triangles (or several triangles all having the same normal). Triangles with a different normal can still share that vertex so long as it isn’t the last vertex.
Allocating normals to vertices can be relatively involved. An arbitrary triangle mesh has roughly twice as many triangles as vertices, so you still end up with roughly two vertices per distinct vertex position (which is better than six, which is the expected value if every triangle has three distinct vertices). However, for quads (pairs of triangles with a common face normal), the ratio is 1:1, and for flat surfaces tessellated into to triangles to deal with non-convexity you may well have fewer distinct normals than vertex positions.
In terms of efficiency, the first option requires generating a normal per-fragment, but this is unlikely to be significant unless you have a lot of overdraw (which can be reduced by rendering the scene roughly from front to back or by using a depth pre-pass to maximise the benefit of early depth test optimisation). The second option avoids this, but if you have more normals than vertex positions increasing the size of the vertex arrays (reducing sharing of vertices) increases the amount of data and the number of vertex shader invocations.