I know one method that used in 3D Studio Max. May be it can be acceptable for you. When 3D Studio Max need to determine sharp and smooth edges of polygons (triangles) they determine angle between normals of such polygons. If angle between normals of two joint triangles is less that some control value (control angle) than such triangles create smooth edge between them, else edge is sharp. By default control angle is 30 degrees.
So you can use dot product of two normalised normals and if value is less than 0.866 (for 30 degree angle) - edge is sharp, else it is smooth.
Yours, Mad Max.