I am looking for an algorithm that can determine the points at the edge of a 3D surface (not essentially in a plane). I have an array of randomly spaced 3D points that define an open surface . I want to get the index of the points at the edge of the surface. Is there a way of efficiently doing the job? I would appreciate any help in this regard.

No you don’t. A set of points is insufficient to define a surface; you also need the topology. Given the topology, determining which points are edge points is straightforward (and only requires the topology, not the vertex coordinates).