TE troubles (add_signed_arb/dot3_arb)

Yeah, I know, chroma keying sucks. But this is emulation, so it has to be.

What I’m trying to do here is to use the alpha test to compare the rgb output (alpha is ignored) of the first texture environment to a color. But it doesn’t work as expected. I have one test scenario where the chroma key is (0,0,0) but instead of making black transparent, I get transparent greens and browns and whatever other colors.

Can you spot something wrong in the following method?

  1. Take color input from texture environment 0, perform GL_ADD_SIGNED_ARB with inverted chroma key color
  2. perform DOT3 product and output to alpha channel
  3. discard pixels with alpha==0

eg, when chroma key is (0,0,0) and input color is (0,0,0) this should yield

  1. (0,0,0)+(1,1,1)-(0.5,0.5,0.5)=(0.5,0.5,0.5)
  2. 4* [ (0.5,0.5,0.5)-(0.5,0.5,0.5) ]^2=0
  3. alpha test fails

The results I get are completely off.

I run this stuff on a Radeon, so I have three texture environments with all the necessary operations supported.
The code is a stripped down and not designed to perform well. The second texture environment is always enabled when executing this and there are always dummy textures bound to the second and third tex environments.

If chroma keying is disabled, I get exactly the expected output. When chroma keying is enabled I do get some gaps here and there but not on the keyed color.

//HACKette part 1
	if (wstate.chroma_enable)
	{
	//set texture environment for chroma keying
		glPushAttrib(GL_TEXTURE_BIT);
		glActiveTextureARB(GL_TEXTURE1_ARB);
			
		glTexEnvfv(GL_TEXTURE_ENV,GL_TEXTURE_ENV_COLOR,wstate.chroma_col_float);
		glTexEnvi(GL_TEXTURE_ENV,GL_COMBINE_RGB_ARB,GL_ADD_SIGNED_ARB);
		glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE0_RGB_ARB,GL_PREVIOUS_ARB);
		glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE1_RGB_ARB,GL_CONSTANT_ARB);
		glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND0_RGB_ARB,GL_SRC_COLOR);
		glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND1_RGB_ARB,GL_ONE_MINUS_SRC_COLOR);

		glActiveTextureARB(GL_TEXTURE2_ARB);
		glEnable(GL_TEXTURE_2D);
		glTexEnvi(GL_TEXTURE_ENV,GL_COMBINE_RGB_ARB,GL_DOT3_RGBA_ARB);
		glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE0_RGB_ARB,GL_PREVIOUS_ARB);
		glTexEnvi(GL_TEXTURE_ENV,GL_SOURCE1_RGB_ARB,GL_PREVIOUS_ARB);
		glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND0_RGB_ARB,GL_SRC_COLOR);
		glTexEnvi(GL_TEXTURE_ENV,GL_OPERAND1_RGB_ARB,GL_SRC_COLOR);

		glAlphaFunc(GL_NOTEQUAL,0.0f);
		glEnable(GL_ALPHA_TEST);
	}

	glBegin(GL_TRIANGLES);
		submit_vertex(converted_vertex[0]);
		submit_vertex(converted_vertex[1]);
		submit_vertex(converted_vertex[2]);
	glEnd();

//HACKette part deux
	if (wstate.chroma_enable)
	{
		//restore combiner pipeline
		glPopAttrib();
	}

edit

Ah, almost forgot, all my tex envs are of course set to GL_TEXTURE_ENV_COMBINE_ARB.

[This message has been edited by zeckensack (edited 03-01-2002).]

I must add, that I use the above to build a stencil mask (with masked off color/depth buffer) and render the actual stuff in a second pass. I stripped too much out, sorry.

Ok, got it. For some strange reason the third texture environment can’t perform the dot product properly. I’ve switched around a few bits and looked at the color output of the chain.

You’d expect a dot product operation to produce grayscale output. Well, it produces funky colors. When I switch the order of operations around (dot product on tex1_ARB, add_signed on tex2_ARB) the output is as expected, albeit useless for my purposes.

The other ops seem to work fine (visual inspection only), but DOT3 is severly broken.

Is this a known issue?
[ul][li]*Radeon DDR 32megs[/li]*Win98SE, official 9009 driver set[/ul]