I currently have a ruberbanded rectangle selection system in my program, by using openGL picking (gluPickMatrix).
I want to implement a polyline selection tool (the user draws a polyline and anything that partially lies inside the polyline gets selected).
My idea is use the code that I already have for the rectangular selection: select from the polyline’s bounding rectangle but “masking” in some way so only the stuff that is inside the polyline is drawn.
My first approach was to use the stencil buffer to mask the area outside the polyline, but I descovered that the stencil does not work in GL_SELECT mode.
So now I am trying to draw the mask on top of everything by using a smaller depth.
However the problem is that I draw part of my scene in ortho mode and part in perspective, so I don’t know how to do the masking the right way with the depth buffer.
Is there any other way to mask a part of the screen that affects the selection?
Or is there any other efficient approach to do this polyline selection?
Any ideas will be welcome.
Thanks in advance,