Need help with spot light pixel shader

Obviously I am too dang stupid to figure something as simple as a spot light via pixel shader, so I am asking for help.

The light source can be anywhere in the view space, position and direction are given by lightPos and lightDir (lightDir is normalized).
I don’t want to pass them via OpenGL’s light stuff. Actually the shader can handle multiple spotlights, but I have simplified it here.
vEye is the view point.
aspect serves for making the headlight round.
cutOff is the opening angle, spotExp the rate of decay.
grAlpha is some global alpha (usually just 1.0)

The OpenGL transformation matrixes are all properly set up (or the renderer wouldn’t work). Some scaling is involved, but that should be stuck in the model view matrix, or am I wrong?

Even if the viewer is the light source for the spotlight, the spotlight doesn’t light the proper area right in front, but is off.
If I rotate the view, the spotlight wanders just somewhere, and I dunno what I am doing wrong.

//base texture and decal
uniform sampler2D btmTex, topTex;
varying vec3 vertPos;
uniform vec3 vEye, lightPos, lightDir;
uniform vec4 matColor;
uniform float grAlpha, aspect, cutOff, spotExp, brightness;
void main(void) {
vec4 btmColor = texture2D (btmTex, gl_TexCoord [0].xy);
vec4 topColor = texture2D (topTex, gl_TexCoord [1].xy);
vec3 lp, ld, lightVec, spotColor, v = vec3 (vertPos.x * aspect, vertPos.y, vertPos.z);
float spotEffect, lightDist, spotBrightness = 0.0;
lightPos -= vEye; //translate
//rotate the light dir by adding it to the light pos, rotating the result, rotating the light pos,
//then subtracting the rotated light pos from the rotated light dir (better way? Cannot figure)
lightDir = vec3 (gl_ModelViewMatrix * vec4 (lightPos + lightDir, 1.0));
lightPos = vec3 (gl_ModelViewMatrix * vec4 (lightPos, 1.0));
lightDir = normalize (lightDir - lightPos);
//this should be standard per pixel lighting fare ...
lightVec = v - lightPos;
lightDist = length (lightVec);
spotEffect = dot (lightDir, lightVec / lightDist);
if (spotEffect >= cutOff) {
	float attenuation = min (brightness / lightDist, 1.0);
   spotBrightness += pow (spotEffect * 1.1, spotExp) * attenuation;
//compute the final light (probably more noob stuff ...)
spotColor = vec3 (max (spotBrightness, gl_Color.r), max (spotBrightness, gl_Color.g), max (spotBrightness, gl_Color.b));
spotColor = vec3 (min (spotColor.r, matColor.r), min (spotColor.g, matColor.g), min (spotColor.b, matColor.b));
gl_FragColor = vec4 (vec3 (mix (btmColor, topColor, topColor.a)), (btmColor.a + topColor.a) * grAlpha) * vec4 (spotColor, gl_Color.a);

The vertex shader:

varying vec3 vertPos;
void main(void)
gl_TexCoord [0]=gl_MultiTexCoord0;
gl_TexCoord [1]=gl_MultiTexCoord1;
vertPos=vec3(gl_ModelViewMatrix * gl_Vertex);
gl_Position = ftransform();

Please be so kind and help me to get this going.

  1. I think there is some problems with your calculations of the light position and light direction. Just treat a light source as a vertex. By the way, you’d better do these calculations in vertex shader.

In vertex shader :

varying ver3 vertPosition, vertNormal, lightPosition, lightDirection, lightVec;
uniform vec4 lightPos; /* lightPos.w = 1.0 */
uniform vec3 lightDir;
vertPosition = vec3( gl_ModelViewMatrix * gl_Vertex );
vertNormal = gl_NormalMatrix * gl_Normal;
lightPosition = gl_ModelViewMatrix * lightPos;
lightDirection = gl_NormalMatrix * lightDir;
lightVec = vertPosition - lightPosition;

However, since the gl_ModelViewMatrix is the mutiplication of modelMatrix( only used to transform verts ) and viewMatrix, if your modelMatrix is not identity, you may need to provide two additional matrices that are only relaive to the view camera to the vertex shader. For example :

uniform mat4 viewMatrix;
unifrom mat3 viewNormalMatrix; /* only the rotation part of viewMatrix */
lightPosition = viewMatrix * lightPos;
lightDirection = viewNormalMatrix * lightDir;

or, doing all the things in world space will make the codes much clearer:

uniform vec4 lightPos;
uniform vec3 lightDir;
uniform mat4 modelMatrix;
unifrom mat3 modelNormalMatrix;
varying vec3 vertPosition, vertNormal, lightVec;
vertPosition = vec3( modelMatrix * gl_Vertex );
vertNormal = modelNormalMatrix * gl_Normal;
lightVec = vertPosition -;

  1. suggestions :

spotColor = vec3 (max (spotBrightness, gl_Color.r), max (spotBrightness, gl_Color.g), max (spotBrightness, gl_Color.b));
spotColor = vec3 (min (spotColor.r, matColor.r), min (spotColor.g, matColor.g), min (spotColor.b, matColor.b));

can be replaced by :

spotColor = max( vec3( spotBrightness ), gl_Color.rgb );
spotColor = min( spotColor, matColor.rgb );

Thanks. Good hint with the min/max calls, didn’t know that. Using .xyz instead of vec3(…) looks much clearer too.

I don’t understand the part about the two additional matrices. The spotlight calculation happens while I am rendering the geometry, and prior to that I am calling glMatrixMode (GL_MODELVIEW), glTranslate(), glScale() and glMultMatrix(). All other matrices have the identity value.

What’s this multiplication of lightDir with gl_NormalMatrix? Where does gl_NormalMatrix get set (I don’t ever set it in my app)?

Actually the more I read your code the less I understand.

Your code doesn’t offer a solution for my problem.

For the simple case that the viewer is the light source, the transformed light direction should always be (0,0,1).

Now if I align the viewer so that it is at position 0,0,0 and facing towards 0,0,1, the light spot is straight ahead of it. But if I move the viewer sideways while keeping the orienation, the spot doesn’t wander with it, but stays where it was (seemingly wandering off to the right).

When I tilt the viewer up 90 deg, the light spot disappears.

The same effect happens when I hardcode lightDir to be 0,0,1. Somehow the transformed vertices don’t seem to be where I expect them.

I have no bloody clue what I am doing wrong.

Edit: Looks like I simply must not subtract vEye from lightPos, because that is included when multiplying lightPos with gl_ModelViewMatrix, right?

Edit 2: Still have a problem if the light source is in some distance from the viewer: Depending on where the viewer faces, the light spot wanders a little with the viewer.

gl_NormalMatrix is part of gl_ModelViewMatrix and is generated by the API, that means, you needn’t set it by yourself.

if gl_ModelViewMatrix is
e00, e01, e02, e03,
e10, e11, e12, e13,
e20, e21, e22, e23,
e30, e31, e32, e33
then gl_NormalMatrix is
e00, e01, e02,
e10, e11, e12,
e20, e21, e22
which is the rotation part of gl_ModelViewMatrix,
while ( e03, e13, e23 )’ is the translation part.

gl_NormalMatrix is often used to transform vertex normals, tangents and so on, since these values won’t be changed by translation.

For more information and details, you could look up in the specification :

False. It is actually transpose of the inverse of that part. This is important if that part contains more than rotations (e.g. scaling).

Thanks for your correction:) In my application, I only pass pure rotations and translations through GL_MODELVIEW matrix mode and never use glScalef to resize models. In this case, the transpose of the inverse of that 3x3 part exactly equals itself and that’s why I didn’t discover that I have made a mistake.

Thanks, but I still don’t know how to fix my headlights …