okapota - In his post, he specifically asked about coplanar polygons, that’s why I brought it up.
The really hard part about degeneracies occurs when they are caused by round off error. In these situations a case may appear degenerate when evaluated one way, but non-degenerate when evaluated another way. You have to be very careful and consistent with your calculations. This is where most of the work in practical computational geometry is done (in my opinion). You read a lot of papers where they talk about an algorithm and the first thing they say is something like “asuuming general position…” Which is fine for discussing an algorithm, but won’t go very far for implementing it.
Chennes - yeah, I’ve read that paper and Simulation of Simplicity is a pretty cool approach for a lot of problems, though I’ve never implemented it. The only problem I see with Simulation of Simplicity is that it might remove some degeneracies that you actually want to see. Like I said though, I’ve never implemented it, so I can’t really say much about it.
Quaternion - Have you done a literature search on boolean operations? I’d be surprised if you couldn’t find plenty of detailed descriptions of the algorithm out there, which would explain exactly how they do the cutting. This seems to me like one of those things that has been done enough times that the chances of coming up with the most efficient solution without standing on the shoulders of others are pretty slim.
You said in your first post that it’s quite easy to cut two polygons which are not coplanar by cutting one with the other’s plane. But what do you do if the polygons only partially intersect? What about cases where only one egdge of a polygon lies in the plane of the other polygon. It’s these kinds of degeneracies that make this a really hard problem.
I’m not trying to be discouraging here, I’m just trying to point out potential issues.