What im wanting to do is if an object is not currently onscreen, is draw a marker on the side of the screen showing which direction the object is in.
currently im doing the following,
Project the objects position into screenspace + see if its with the viewport
if its not depending on what the x+y number returned back from the Project(…) function to see what plane to test against + find the intersection point
Test if the bounding box is visible, if so, goto 2 else goto 3.
Draw model, goto 6.
Transform object center (Oc) to eye space (Ec)
Construct a plane from viewing direction and Ec
P = crossproduct(vec3(0,0,1),Ec)
plane P is defined by all points where Pxx+Pyy+Pz*z=0
Find intersection points of this plane with the line segments (or only the relevant one like you said) at the near plane (left,right,bottom,top and near coordinates of your glfrustum calls)
Intersections are found by using parameterization of the line segment e.g. line is defined as