Okay, so to be clear, you are rendering a 3D model to the screen and you want to know which points in (x pixels, y pixels) on your screen fall on your mesh and which don’t? Are the points in screen space (x pixels, y pixels) or 3D space like your model?
One easy way to do it is using the stencil buffer.
- make sure your stencil buffer is on and working
- clear it to something like 0x0
- render the mesh setting the stencil values to 0x1, where the mesh get’s rasterized
- render your points to the screen setting the stencil value back to 0x0
- For each point, call glReadPixels() on your stencil buffer reading in a single pixel (width=1,height=1). If that value is 0x1, it lies inside the mesh. If it is 0x0, it doesn’t.
This assuming your points are in 2D. It’s very slightly more tricky with 3D points depending on what you’re trying to do.