Selecting inside a polyline

I currently have a ruberbanded rectangle selection system in my program, by using openGL picking (gluPickMatrix).
I want to implement a polyline selection tool (the user draws a polyline and anything that partially lies inside the polyline gets selected).
My idea is use the code that I already have for the rectangular selection: select from the polyline’s bounding rectangle but “masking” in some way so only the stuff that is inside the polyline is drawn.
My first approach was to use the stencil buffer to mask the area outside the polyline, but I descovered that the stencil does not work in GL_SELECT mode.
So now I am trying to draw the mask on top of everything by using a smaller depth.
However the problem is that I draw part of my scene in ortho mode and part in perspective, so I don’t know how to do the masking the right way with the depth buffer.

Is there any other way to mask a part of the screen that affects the selection?
Or is there any other efficient approach to do this polyline selection?

Any ideas will be welcome.

Thanks in advance,

What is wrong with just using picking as you do now, when the users klicks on any line segment the polyline is selected.

If you have a closed polyline and what to allow clicks “inside” it you could render it as a filled polygon instead when in selection mode and you get the same effect.


mikael_aronsson, I think you got it wrong, the OP wants to select objects within a polyline, not the polyline itself.

Ooops, sorry.

Well, then it does get a little more tricky, “a point in polygon” algorithm is not very difficult to find, but then you need to check against the 2D coordinates from the hit, and you don’t have them.

I guess you could create an (you would need more then 1 actually) “inverse” polygon(s) between the rectangle and the outside of the polyline, if you include that in the picking you could compare if an object hits that polygon(s) or not, does that make any sense ?