for a 3D scene, you need depth testing, NOT stencil testing. besides that, you need to generate a mesh for the cube with holes in it. if you try to “trick” something together with stencil test etc (e.g. filling the stencil buffer with the holes, and then draw the cube etc) good luck with that …
This isn’t a “trick”. This is what stencil testing is meant for. The reason why glStencilOp has increment/decrement/invert operations is for counting the number of times a ray intersects a surface, i.e. testing whether the endpoint is inside or outside the surface.
But the first part is wrong; it can’t be done this simply. If you look at the left-hand side, you’ll notice that the sphere doesn’t just cut out the part of the face which is inside the sphere, but also the part which is behind the part of the sphere which protrudes.
For that: draw the front faces of the cube first, writing to the depth buffer (there’s no need to enable depth testing; the front faces of a convex object cannot overlap each other). Then draw the the sphere, both inside and outside, testing against (but not writing to) the depth buffer, inverting the stencil value when the depth test passes.
Once you’ve done all of that, the stencil buffer will contain non-zero for the parts of the cube which are inside the sphere (i.e. all fragments for which a ray between the eye and the nearest point on the cube intersects the sphere an odd number of times).
You can then do the same for the inside faces of the cube (you can’t do both at once, as the outside faces will obscure the inside faces).