Need advice on own windowing toolkit


I am currently working on writing my own windowing toolkit. I do this just for fun and for the honor; I want to learn more about OpenGL and programming in general and I think this will be learning a lot throughout this project.

Unfortunately I ran into a little problem and I would like to ask for some help on this. I decided to use a recursive structure for the GUI components; they are nested into each other like a tree.
GUI example:

GUI as a component tree:

My problem is that I need to do some clipping in the GUI. The child of a component should never extend over the boundary box of its parent component. If I have a window and some buttons inside the buttons should not be drawn outside of the window if the window is resized to be very small.

I personally know of 2 ways of doing this clipping; scissor-test and stencil-test. I tried with scissor test and it works so far, but my problem is, that I try to store all the rendering data for all components in one big VBO, each time I want to apply a scissor test I have to make another draw call. If the component tree is very high instead of wide this would result in awful performance I guess.
I haven worked too much with stencil tests yet, but I thought these could potentially make it a little better.

So, after that wall of text, these are my questions:

  1. Would using stencil test be a better idea for clipping? (And why?)
  2. If stencil tests would be better, can you give me an advice on how to use them properly for this task?
  3. Is there some other way I could do this that I just dont know about?

Thank you very much for the answers, I really appreciate the help.

They both operator at the fragment level. The main difference is that a stencil can be any shape and is ideal for things like HUD’s whereas the scissor test is restricted to a rectangle. So if your clipping is only a simple rectangle I would expect the scissor test to be faster as you don’t have to create the stencil.