I would like to position the camera in the scene looking at a model and determine which triangle is the center pixel - ie. if i drew a ray from the camera to the model which triangle would it intersect?

Is there a way to do this without actually producing a window and displaying the object (what is usually done from the display() function called by glutMainLoop() )?

right - but I have 10,000 triangles and I want to check that intersection with 10,000 rays - this is insanely computationally expensive, because I have to intersect each ray with each triangle and then see which intersection was closest. If I used opengl, it would determine a handful of triangles which are “straight ahead” by using the clipping planes and then I could only intersect each ray with 5-ish triangles instead of 10,000.

I don’t think it is an efficient solution. A software one is to use better data representations like kd-tree that improves enormously intersection search. This is used in most of ray-tracers.

clipping 9995 out of 10,000 triangles in hardware is clearly better than using a kd-tree if you ask me, and the code is much much simpler

I must miss something, but the way you said it was that you find which ray intersection is closest from the eye but before that, how do you find all possible intersection between the ray and triangles using Opengl? Is there any paper, link you are working from?