Help with luminance alpha masking

[primal_visions]

hey,

im trying to take the luminance of one texture and set the inverse luminance to the alpha of another texture for alpha masking but im having problems.

with this i get just a slightly alpha’d out whole texture mixed with the layer below, not the hole im looking for. if i add the textures together beforehand it mixes the alpha’s together as well.

anyone have a clue what i can do and why this doesnt work?

after i come out of the shader i use glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA );

cheers, james =)

uniform sampler2DRect mask;
uniform sampler2DRect masked;

void main(void)
{
vec4 originalColor = texture2DRect(mask, gl_TexCoord[0].xy); //the mask
vec4 maskedImage = texture2DRect(masked, gl_TexCoord[0].xy);
vec4 mixed = vec4(0,0,0,0);

vec4 LUMINENCE_FACTOR  = vec4(0.27, 0.67, 0.06, 0.0);

// take the dot product of the luminance factor
float lum = dot(originalColor, LUMINENCE_FACTOR);

lum = 1.0 - lum;

mixed = vec4(maskedImage.rgb, lum);

gl_FragColor = mixed;   

}

[NiCo1]

Do you get black color values for the hole if you disable blending and output glFragColor = vec4(lum,lum,lum,lum)?

N.

[primal_visions]

err yes i get black, i had it on an inverted image for a sec which confused me, but i still can’t seem to affect the alpha values properly with my luminance values. it seems to ignore me for the most part…

[NiCo1]

And your blend equation is set to GL_FUNC_ADD?

N.

[primal_visions]

it was set to that by default but i set it to that in code but it hasn’t changed anything.

i would expect to see some of the image taken out where my mask is but it seems to just apply a little bit of alpha to the whole image.

if i just set the alpha of my mask to the alpha of my layer it works fine on grayscale images or pics with alpha.

it would just be a bit better if it worked off of luminance instead of only alpha then it would work on rgb images as well for masking, i just dont understand why i cant do that when it is nearly the same thing and i can see that the luminance values are correct.

if i use those luminance values on itself it uses them correctly. just not on the other layer.

should i be able to do this?

thanks for your help =)

[NiCo1]

I just plugged in the shader into one of my projects and it seems to work just fine. originalColor is taken from a black rgb texture with a white circle and I can see a circular hole when rendering…

N.

[primal_visions]

hmm, i guess its something to do with my main program. im on a mac on an ati card so things might be a bit different.

so odd that it would work with the masks alpha values and not the luminance values even though they are the same for the image im using. as i said it just ignored my values and put an alpha of about 0.4 across my whole image instead of the hole.

thanks for letting me know it works though and that i can do that =)

i thought it should work, it just didnt make sense otherwise.

were you using the same blending mode i was?
glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA );

anyclue what might cause it to work with alpha values and not my luminance values for the alpha on my machine?

cheers, james =)

[NiCo1]

You’re welcome and yes, I used the same blending mode. From what you say, it looks like a driver bug. Maybe a driver update can fix the problem?

@primal_visions

so odd that it would work with the masks alpha values and not the luminance values even though they are the same for the image im using.

Do you mean that it works if you use:

vec4 LUMINENCE_FACTOR = vec4(0.0, 0.0, 0.0, 1.0);

?

N.

[primal_visions]

yes it works like that =)

what does that mean?

[primal_visions]

oh and there are no driver updates with macs… only when they bring out a new patch to osx…

[NiCo1]

If you’re 100% sure that

0.27originalColor.r + 0.67originalColor.g + 0.06*originalColor.b == originalColor.a

then it’s a bug and I would suggest a driver upgrade/downgrade.

N.

Edit: oops, just read your update comment…

[primal_visions]

well im not sure of the exact values but in the grayscale image im using they are going to be roughly the same or inverted, i can never remember which way they go but even if invert them it just applies the inverted alpha across the whole image. and in the end its the coefficients that determine the values and there are a bunch you can use for luminance, but i would have thought they balanced out and ended up roughly the same.

the only thing i can do is try it on other hardware and another program =) thanks again for your help and see ya =)

[NiCo1]

Good luck!

Maybe you can get an update from AMD’s driver page

See ya

N.