# Calculating normals for animated deformation

Hello, everyone!
Nice to be here.

I’m still pretty new to the OpenGL graphics pipeline, but skilful enough to comfortably swing around VS, FS, FBOs and a few rendering methods with a up to a few model count.

I’ve gotten to a point where I’m interested in animated mesh deformation. The mesh could be animated by texture displacements, by modifiers(bend, taper, etc.) or some other math operations.

The problem is that after I’ve done my deformation I don’t have a clear understanding how to recalculate normals? Methods I have tried gave an unwanted faceted look.

Two deformation example scenarios would be (both done in VS):
- A plane displaced by an animated noise.
- A cylinder bending into a C form.

In the OGL super bible such a topic is not covered. From all of my web searches I’ve found the term “per vertex normals” from lighhouse3D articles. (Can’t include link) But the article suggests averaging out face normals that share the vertex.

It seems as a pretty good way to go. The key to use this method involves knowing the data of neghbouring faces, so I looked at the geometry shader, but it does not have a function that returns the neighbour data. There a few other hints around the web that suggest that says it’s somehow possible, but no guide at all.

Could someone describe a bit better what I’m searching for or maybe guide in the right way?

While writting this I thought that it could be possible to store the neghbouring face data in an attribute, but it seems a bit heavy for one vertex to carry and manipulate extra data that is already out there.

Arnis

You can get access to some of the neighbouring vertices by using one of the primitive modes GL_TRIANGLES_ADJACENCY or GL_TRIANGLE_STRIP_ADJACENCY. This requires a change to the structure of the element array (you need to pass 6 vertices per triangle or 2(N+2) vertices for an N-triangle strip). But you only get the vertices for the triangles which share an edge with the current triangle, not those which only share a vertex. See §10.1.13 and §10.1.14 of the OpenGL 4.6 specification for more details.

If you want access to all of the neighbouring faces, the best option is likely to be a compute shader. I’d probably use two passes: one to compute the face normals, and one to compute the vertex normals from the neighbouring face normals. The first needs access to the element array (typically as a SSBO) and would have one invocation per face, the second needs the inverse mapping (i.e. the set of faces adjacent to each vertex) and have one invocation per vertex.

If you’re deforming the mesh by a smooth (or at least C1-continuous) function, the Jacobian matrix gives you the linear transformation at a point. Normals should be transformed by the inverse-transpose of that matrix to ensure that they remain normal to the surface. The computational cost depends upon whether you can obtain a simplified expression for the inverse (although you can ignore the scale factor if you’re going to end up normalising the normals) or whether you have to evaluate 9 partial derivatives directly then invert the resulting matrix.