(non realtime) Hidden face removal

I’ve got a few 3d-models which aren’t drawn very cleanly - that is: There are some faces which are never visible from any point of view - as for example two intersecting spheres.
Having given up googling for an answer I was searching for a clean way to remove the hidden faces but didn’t come to a “clean” (that is: mathematically correct) solution.
Such a solution would mean to find all triangles for which it is impossible to cast a ray from anywhere of it’s surface to a point sufficiently far away without intersecting any other triangle.
Having thought about this this would form an (in-)equation system that contains different equations as well as inequalities: First one knows the ray is originated on the triangle to test for invisibility:

P = A + a*AB + b*AC  , a>=0, b>= 0, a+b <= 1

and the ray cast from it

R = P + c * D

may not intersect with any triangle

T = Q + s*S + r*R, s+r <= 1, s>=0, r>= 0

which leads to:

R != T, for any (valid) a,b,c,D,T (in all other triangles)


Can this be solved efficiently? Is there software performing perfect non-realtime face removal?

If you know that the visible faces that “hide” the invisible faces form a closed mesh then the problem is simply reduced to a “point in concave primitive” issue which you’ll be able to easily find efficient algorithms for. If the mesh has “holes” then it is a bit trickier though, but definitely solvable efficiently.

Hmm. Is there software - eg. Modeling-Tools - available that can do such model-cleanup?

EDIT: It has been some time since I looked into the test-Version of 3DS-Max and that was not a feature I searched for.

You could use Bullet Physics along with its included convex hull decomposer.