Yeah, I know, chroma keying sucks. But this is emulation, so it has to be.
What I’m trying to do here is to use the alpha test to compare the rgb output (alpha is ignored) of the first texture environment to a color. But it doesn’t work as expected. I have one test scenario where the chroma key is (0,0,0) but instead of making black transparent, I get transparent greens and browns and whatever other colors.
Can you spot something wrong in the following method?
- Take color input from texture environment 0, perform GL_ADD_SIGNED_ARB with inverted chroma key color
- perform DOT3 product and output to alpha channel
- discard pixels with alpha==0
eg, when chroma key is (0,0,0) and input color is (0,0,0) this should yield
- (0,0,0)+(1,1,1)-(0.5,0.5,0.5)=(0.5,0.5,0.5)
- 4* [ (0.5,0.5,0.5)-(0.5,0.5,0.5) ]^2=0
- alpha test fails
The results I get are completely off.
I run this stuff on a Radeon, so I have three texture environments with all the necessary operations supported.
The code is a stripped down and not designed to perform well. The second texture environment is always enabled when executing this and there are always dummy textures bound to the second and third tex environments.
If chroma keying is disabled, I get exactly the expected output. When chroma keying is enabled I do get some gaps here and there but not on the keyed color.
//HACKette part 1 if (wstate.chroma_enable) { //set texture environment for chroma keying glPushAttrib(GL_TEXTURE_BIT); glActiveTextureARB(GL_TEXTURE1_ARB); glTexEnvfv(GL_TEXTURE_ENV,GL_TEXTURE_ENV_COLOR,wstate.chroma_col_float); glTexEnvi(GL_TEXTURE_ENV,GL_COMBINE_RGB_ARB,GL_ADD_SIGNED_ARB); glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE0_RGB_ARB,GL_PREVIOUS_ARB); glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE1_RGB_ARB,GL_CONSTANT_ARB); glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND0_RGB_ARB,GL_SRC_COLOR); glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND1_RGB_ARB,GL_ONE_MINUS_SRC_COLOR); glActiveTextureARB(GL_TEXTURE2_ARB); glEnable(GL_TEXTURE_2D); glTexEnvi(GL_TEXTURE_ENV,GL_COMBINE_RGB_ARB,GL_DOT3_RGBA_ARB); glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE0_RGB_ARB,GL_PREVIOUS_ARB); glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE1_RGB_ARB,GL_PREVIOUS_ARB); glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND0_RGB_ARB,GL_SRC_COLOR); glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND1_RGB_ARB,GL_SRC_COLOR); glAlphaFunc(GL_NOTEQUAL,0.0f); glEnable(GL_ALPHA_TEST); } glBegin(GL_TRIANGLES); submit_vertex(converted_vertex[0]); submit_vertex(converted_vertex[1]); submit_vertex(converted_vertex[2]); glEnd(); //HACKette part deux if (wstate.chroma_enable) { //restore combiner pipeline glPopAttrib(); }
edit
Ah, almost forgot, all my tex envs are of course set to GL_TEXTURE_ENV_COMBINE_ARB.
[This message has been edited by zeckensack (edited 03-01-2002).]