Object selection

In OpenGL’s selection mechanism,We can use Mouse to picking graphic object.In the red book, it use gluPickMatrix to set the select area and return hit records by " hits = glRenderMode (GL_RENDER); " . But if i want to use a non-rectangle area to pick graphic objects,or use a closed polyline to pick graphic objects,how could i do it smart?


It may be possible to do non-rectangular selection with the stencil buffer. I’m not sure I understand the question?

In red book,it set “glRenderMode(GL_SELECT);” and then drawn graphic,set gluPickMatrix, finally set “GLint hits = glRenderMode (GL_RENDER);”. it return hit records int the “GLint hits”.I don’t understand how it really work in detail.I just know I can set the hit coordinates with hit width and height in gluPickMatrix. But it is a rectangle area for picking.I want to write a more power picking function to select graphic objects not only pick by using a point or a rectangle area.I hope to use closed polyline to set a area and use this area to pick graphic object.I know a stupid way to write this, find the area in pixel coordinate and repeatly do the gl select mechanism in a pick point.But everytime i do it once,i need to draw all graphic once.If the area have N pixels it draw N times.This wasts time.

“It may be possible to do non-rectangular selection with the stencil buffer”, i dont understand how to use stencil buffer to do this function, could you give me more hint,
Thank you.