Billboards and geometry shader frustum culling

I’ve run across a conceptual problem using geometry shaders to perform frustum culling. Here’s how it works:

  1. Render every possible instance in a culling pre-pass, using the geometry shader to discard instances outside of the camera frustum. The resulting index IDs are rendered into a vertex buffer and a PRIMITIVES_DRAWN query is used to determine the number of instances that are visible.

  2. In a second pass actually draw the geometry, using the results of the query to get the number of instances, and the vertex buffer to store the ID of each instance.

That all works extremely well. Now I want to add billboarding, by drawing a quad for every instance that is over a certain distance away, and I suddenly realized this is very difficult. Other than rendering a second pre-pass, is there any way you can think of separating the billboard and non-billboarded instances? I mean, I can detect them easily enough in the culling pre-pass, but actually rendering those two things separately is a problem. How can I output the billboard instances into one buffer, and the model instances into another, and return a count for both?

Any ideas?

What OpenGL version are you targeting? Writing to two SSBOs (shared storage buffer objects) and two atomic counters is one way, if you have GL4.3 support. You could also use a compute shader for GL4.3 instead of a vertex/geometry shader.

GL4.0 also has geometry shader streams, so you could output IDs to two different streams with an invalid ID for the buffer that shouldn’t render (a primitive restart index, for example). Though I’m not sure how you’re doing the non-billboarding render, so that may not work.

[QUOTE=malexander;1279378]What OpenGL version are you targeting? Writing to two SSBOs (shared storage buffer objects) and two atomic counters is one way, if you have GL4.3 support. You could also use a compute shader for GL4.3 instead of a vertex/geometry shader.

GL4.0 also has geometry shader streams, so you could output IDs to two different streams with an invalid ID for the buffer that shouldn’t render (a primitive restart index, for example). Though I’m not sure how you’re doing the non-billboarding render, so that may not work.[/QUOTE]
OpenGL 4.0 compatibility is the requirement.

What I am trying to avoid is unnecessary vertex processing in the final draw pass. Although an invalid ID could be used to discard instances, the damage would still be done because the GPU would have to plow through the vertex shader for each visible instance, with the full model geometry.

I set it up to render in two pre-passes, with two vertex buffers and two different queries, but if anyone has any bright ideas that are 4.0-compliant, I am all ears.