Need help with NV_transform_feedback

I am trying to use the NV_transform_feedback extension in OpenGL to expand a compact description of rectangles and trapezoids to one that is directly renderable by OpenGL. For the shaders that will render the primitives, a lot of data needs to be replicated across all vertices of a primitive, and I am using the texture coordinates for this.

Feeding a transform feedback shader with GL_POINTS and expanding each of these to 8 vec4 varyings works just fine, and allows transfer of rectangles to GPU memory with minimal latency. But for the trapezoid structures, I need to output even more varyings. However, trying to output 12 vec4 varyings results in a GL_INVALID_OPERATION when calling glDrawArrays to do the expansion with the transform feedback shader. And calling glDrawArrays too many times even makes the machine lock up, forcing a hardware reboot. More than enough space has been allocated in the VBO to hold the recorded data.

In the specs of the NV_transform_feedback extension, it says that MAX_TRANSFORM_FEEDBACK_INTERLEAVED_COMPONENTS_NV gives you the maximum numbers of components that can be output by a transform feedback shader. This is reported as 16 by the 100.14.19 Linux driver and as 64 on the 169.04 and later drivers. But no matter which driver I use, the result is the same.

I have spent much time trying to figure out the reasons for the error but still cannot make any sense of why it does not work as expected. Any help on this would be much appreciated.

Perhaps instead of outputting more varyings, you could use a geometry shader and output multiple primitives under one varying with transform feedback?

One of my colleagues at NVIDIA pointed me at this. I can’t see anything obviously wrong from your description.

It sounds like your basic approach works, since you can capture 8 vec4’s in interleaved mode successfully. 12 vec4’s should work just fine, too. Am I correct to assume the only difference between the working and failing case is the number of vec4’s passed out by the vertex shader?

The GLSL implementation in very old NVIDIA drivers for the GeForce 8 series (which didn’t support most of the new GLSL extensions) did not take advantage of all hardware features and may not have been able to output 12 vec4’s at all, let alone with transform feedback. If you still have the same problem with 169.04+, that is definitely not the issue.

One other thing, probably not relevant in your case, but just to be sure… MAX_TRANSFORM_FEEDBACK_INTERLEAVED_COMPONENTS_NV is the number of components that can be captured in interleaved mode during transform feedback; the shader could conceivably emit more components. 64 components should be the right value for the GeForce 8 series. I vaguely recall a bug where were erroneously reporting wrong (lower) limits in older drivers. Vertex shaders can only push out 64 components total (including position), so if you grab 48 for transform feedback, only 16 are available for other purposes. (The 48 components that you grab in this case can go to the raster, too, if rasterization is enabled.)

With the INVALID_OPERATION, it sounds like the driver found something wrong. But I’m not sure what it will turn out to be – these errors are sometimes hard to find. My first suggestion would be to make sure that your program with 12 vec4’s actually linked successfully – check the status and info logs.

If you have something that shows the problem I can build and run, that would probably be the best way to figure out what’s going on. It could well be a driver issue. My email address is pbrown at