Hi,
This question is about how OpenGL, glxSwapBuffers and glxMakeCurrent operate when there are several 20 fps videos being fed, each one to their own X windowses.
My scheme goes (omitting some detail) like this:
- memcpy to PBO memory addresses (there are several threads doing this). There is a “pool” of PBO objects that is being constantly re-used.
The following steps are performed by a master thread (only thread that touches OpenGL):
- PBO => texture “swap” at the GPU (each video stream as its own texture set)
[highlight=c++]
// y
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, y_pbo);
glBindTexture(GL_TEXTURE_2D, y_tex); // this is the texture we will manipulate
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, w, h, format, GL_UNSIGNED_BYTE, 0); // copy from pbo to texture
glBindTexture(GL_TEXTURE_2D, 0);
// u
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, u_pbo);
glBindTexture(GL_TEXTURE_2D, u_tex); // this is the texture we will manipulate
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, w/2, h/2, format, GL_UNSIGNED_BYTE, 0); // copy from pbo to texture
glBindTexture(GL_TEXTURE_2D, 0);
// v
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, v_pbo);
glBindTexture(GL_TEXTURE_2D, v_tex); // this is the texture we will manipulate
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, w/2, h/2, format, GL_UNSIGNED_BYTE, 0); // copy from pbo to texture
glBindTexture(GL_TEXTURE_2D, 0);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0); // unbind // important!
glBindTexture(GL_TEXTURE_2D, 0); // unbind
// glFinish(); // maybe not use this (see below)
3) Drawing the image. Each video has its own window_id, VAO, texture set etc. Drawing goes approximately like this:
[highlight=c++]
glXMakeCurrent(display_id, window_id, glc) // each video in its own X-window ..
glViewport(0, 0, x_window_attr.width, x_window_attr.height);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); // clear the screen and the depth buffer .. this can be commented out
shader->use(); // use the shader
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, y_index);
glUniform1i(shader->texy, 0); // pass variable to shader
glActiveTexture(GL_TEXTURE1);
glBindTexture(GL_TEXTURE_2D, u_index);
glUniform1i(shader->texu, 1); // pass variable to shader
glActiveTexture(GL_TEXTURE2);
glBindTexture(GL_TEXTURE_2D, v_index);
glUniform1i(shader->texv, 2); // pass variable to shader
glBindVertexArray(VAO);
glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, 0);
glBindVertexArray(0);
if (doublebuffer_flag) {
std::cout << "RenderGroup: render: swapping buffers "<<std::endl;
glXSwapBuffers(display_id, window_id);
}
I have timed all OpenGL operations and it seems that the problem is in GLX
A) A Single video - works nicely
[ul]
[li]PBO=>TEX takes systematically 3-4 ms [/li][li]Swap buffer might alert every… 10 seconds or so, that glxSwapBuffers takes around ~ 3 ms … but most of the time its less [/li][li]However, if I start to resize the window, then it might take 5+ ms. [/li][li]… btw, why is this? The window system blocks glxSwapBuffers? [/li][/ul]
B) Two videos - doesn’t go so smoothly …
[ul]
[li]glxSwapBuffers takes constantly 4+ ms [/li][li]glxmakecurrent starts to take sometimes 10+ ms (quite sporadically) [/li][li]… these have consequences to PBO=>TEX as the whole OpenGL pipeline seems to stall [/li][li](Late frames are dropped and not being PBO=>TEX’d & rendered … so there is no saturation from that) [/li][/ul]
With more videos, the whole thing just stalls.
What am I doing wrong here…? Please, help me understand the following issues:
When I have two videos, there can be two frames coming immediately one after the other (one from camera 1 and the other from camera 2). I’m actually testing with two streams coming from the same multicast source, so the frames arrive at almost identical times. Because of this, we can have:
[ul]
[li]Rapid consecutive glxMakeCurrent calls [/li][li]Rapid consecutive glxSwapBuffers [/li][li]etc. [/li][li]There is an individual YUV texture set for each stream, so textures will be overwritten at each 40 ms or so … enough time for them to get rendered I guess. [/li][/ul]
OpenGL should just queue these requests … right? How about the glx calls … they do some blocking?
For example, if I take out that glFinish() call, then the PBO=>TEX part exits immediately, is queued by OpenGL and executed … when? At the latest, when glxSwapBuffers gets the next screen refresh? Removing all glFinish calls does not seem to help things.
Regards,
Sampsa
Something I’ve found:
https://www.khronos.org/opengl/wiki/Swap_Interval#In_Linux_.2F_GLX
https://www.gamedev.net/forums/topic/584025-glxswapbuffers-very-slow-potential-problems/
It was, after all, about glXSwapInterval.