If your object is just an arbitrary triangle mesh, then I don’t think there is a way that’s fundamentally better than checking each edge.

But you can optimize by calculating which edges can theoretically be part of the outline. Edges where the object is concave or flat can never be part of the silhouette, so they need not be checked. This can be precalculated and stored with the object.

Let’s say you have these two polygons, and you want to know if the mesh is concave/convex/flat at the shared edge.

B
/|\
/ | \
A | D
\ | /
\|/
C

You just have to check the position of D relative to the plane defined by ABC. If it is on the plane, the edge B-C is flat. If D is below the plane, B-C is convex, if D is above the plane, B-C is concave.