I can’t see why you would need to compute a triangle normal unless you’re generating mesh data. That is, you’re not loading per-generated data. After all, most transforms on mesh data can transform normals just as well as positions (though you may need to provide a special matrix for doing so). So if your mesh was built off-line, then whatever tool which produced that mesh can also produce normals for it.
So let’s assume you have some process executing during your program that is producing position data for a mesh out of whole cloth.
Well, lots of such processes can generate vertex normals along with vertex positions. Vertex normals can be generated for bicubic patches for example with some creative mathematics in the generation process. Subdivision surface and other mechanisms also tend to be able to generate normals alongside positions.
However, if your mechanism for building such positions does not allow for generating normals along with positions, then you’re going to have to compute those normals from the mesh itself. This requires iterating over the triangles around a position and taking the average of the normals for the adjacent triangles (an average weighted by the surface area of each triangle).
You can’t do that in a geometry shader, as there can be quite a few triangles adjacent to a single vertex. Indeed, it’s probably not reasonable to do it in a compute shader either. So if you used an on-GPU process to generate the positions, you’re going to have to read that back on the CPU (including topology data) to compute vertex normals.
If all you want is a per-face normal, you don’t need to compute that in the GS at all. Just compute it in the fragment shader.
Pass the position of the vertex from the VS to the FS. Note that the space of the vertex is important; transform the position into whatever space you want to use for lighting. Then use the FS functions
dFdy on this position. These functions compute the rate-of-change of the position in the window-space X and Y directions. Of course, the rate-of-change is basically just a direction pointing along the window-space X and Y directions of the triangle. If you take the cross product X cross Y, and normalize the result, you get the triangle’s normal. And you’ll get the same value at every point on the triangle, since the rate-of-change of a value being linearly interpolated is fixed.