Frame buffer color keying


I need to replace parts of the current frame buffer that have a specific color with a texture. In other words, I have a background texture which should “shine through” all black (or green, or blue) parts of the scene. So basically I need some sort of destination color keying. I cannot use custom pixel shaders, since the hardware does not support it.

I was hoping this is directly supported by OpenGL, but the only way I found is actually retrieving the whole buffer using glCopyPixels, modifying it manually, and storing it back using glDrawPixels, which of course is very slow.

Background: I’m working on an augmented reality extension to an existing (but black-box) rendering engine. I get a camera image from an HMD which should be enhanced with cg 3D models. These models should be realistically occluded by real world objects (like walls), so they’re drawn into a model of the real world. Now the problem is replacing the walls by the camera image.

This is very easy if one has access to the engine source: Clear the frame buffer with the camera image, render the walls to the stencil buffer only, and then render the visible models on top of it using stencil testing. Sadly, I do not have access to the engine source, so this is not the way to go. The basic layout of the code is as follows:

<free to change this part>
engine.paint(); // paints world and objects
<free to change this part>

If you have any other ideas on how to do this, or know of a more elegant way to solve the problem, please let me know.


This depend on what control you have on the engine itself (and what state changes it uses internally)

I assume from the above that you can set the color of the walls rendered?

If the engine does not use/set alpha blending/alpha testing, what you could do is this:
-Ensure you have destination alpha render surface
-Set the walls with a alpha value of 0 (ensure alpha testing and blending is off and all other surfaces have an alpha of 1)


-After the render only the walls have a alpha of 0 so enable blending with ONE_MINUS_DST_ALPHA:ZERO and render a full screen quad with you camera image.

If you can also try this old thread:;f=3;t=007658
(assuming you have two texture support and a stencil mask -see the last post by zeckensack) You may also have to turn off color buffer writes when setting up the stencil buffer.

Thanks a lot for the fast and great answer! This (“abusing” the alpha channel) works perfectly and at top speed. I love this board, no bull****, just qualified and precise answers. Thanks again!