If this is your use case, then I really don’t understand why you’re even considering clipping against the side frustum planes.
The way I might implement it: You’ve got a tiny little selection box on the screen. When they click the mouse, you’ve got to go figure out the closest thing that it “hit”. I’d immediately then build a vector starting at the near clip plane centered inside the selection box and pointing parallel to the -Z axis in eye space. Then I’d intersect this against your scene.
If you’ve still got the saved “draw list” from your previous cull, you could look through that, though it could be huge. Instead, I’m assuming you have a spatial acceleration data structure for your scene (bounding volume heirarchy [BVH] or spatial subdivision structure) of some sort which you use for cull. If so, I’d just “intersect” your ray against your scene using it. That should be fairly efficient. You just keep the closest intersection point/object.
With this, you don’t even care about clipping against your side (or near) frustum planes at all. The only frustum plane you possibly might care about is your far clip, because if the nearest hit is beyond the far clip then obviously the ray didn’t hit anything in the frustum.
A variation on this is to due full quad-frustum clipping on the sides of your selection box, keeping the closest hit anywhere inside the box. More expensive.