this sounds similar to work I have done and to a certain extent to the idea I’d like to implement but need to sort out the minimisation problem first (if, indeed, I can be bothered).
I’ll describe the approach I did to see if it makes sense for in your case. The project I am working on involves stacking objects on top of each other. The metaphor I use is something like placing building blocks in the scene.
The engine always has some concept of the ‘ground plane’ somewhere in three-space (not necessarily the XZ plane—or, indeed, any axis-aligned plane) and the user is able to click on some point on the screen and drag out a new point. The idea is that the first mouse click is back-projected to intersect with the ground plane: this forms a 3D “anchor point” in world space and a 2D reference point on the plane. When the mouse is dragged, a new point is similarly back-projected to the ground plane. This new point forms a reference point and I compute a transformation wiht two degrees of freedom to map the anchor point to the origin of the object I want to place and the reference point to some arbitarily defined anchor point.
In a simple example, suppose I have the ground plane x-1=0 and I clicked an anchor point <1, 2, 3> (this pt satisfies the plane equation) and dragged a new reference point <1, 3, 3> for some geometry with a local origin <0, 0, 0> and handle <1, 0, 0> then the transformation I compute for this configuration is to map <0, 0, 0> to <1, 2, 3> AND map <1, 0, 0> to <1, 3, 3>… which would give me am effective transformation… well, it’d rotate 45 degrees around the object’s y axis, a unit scale and a some other planar transformation to map the XZ plane to the x=1 plane.
Anyway. I do all that without rendering anything to the screen because its all just simple solving linear equations: you know the ray you’ve formed by back projecting a pixel in three space and you can easily compute the intersection of that ray with the plane equation by formulas available on the web, somewhere (like mathworld.wolframm.com). The only tricky aspect is computing the transformation along the plane, but if you keep your parameters simple enough then there WILL be a uniqye set of parameters that will exactly fit. You can’t use all parameters; pure rotation about two axis, for example, won’t span the entire space on the plane: you’d have to have one rotation and a scale or a translation to ensure that all points on the plane WILL have valid parameters, otherwise you’ll need to do some minimisation stuff.