Post whole shader. You can use sampler2DShadow and shadow2DProj and result is 0 or 1 dependeing if fragment are lited or not. Use more samples and sum and divide results win number of samples.
This is one of your first shaders? Right?
There is a lot of error. Correct shader should look like:
// vp
// *attribute* are per vertex attribute. you have only one light so use uniform
uniform vec3 lightpos; // light position in worldspace
uniform vec3 lightnorm; // light vector in worldspace
// use varings to pass variables
// between vp and fp. it will interpolated
varying vec4 projShadow;
varying float LightIntensity;
void main()
{
// calc worldspace vertex position and normal
vec4 realPos = gl_ModelViewMatrix * gl_Vertex;
vec3 realNorm = gl_NormalMatrix * gl_Normal;
// assume lightpos are in world space
float lightdist = length(realPos.xyz - lightpos);
LightIntensity = dot(lightnorm, realNorm) * lightdist/400.0;
// assume light mating in gl_TextureMatrix[2]
projShadow = gl_TextureMatrix[2] * realPos;
gl_Position = ftransform();
}
//fp
// a shadowmap. Use this kind of sample for shadow map
uniform sampler2DShadow shadowMap;
// a spotlight texture. Usualy a white filled smooth cicrcle on black background
uniform sampler2D spotlight;
varying vec4 projShadow;
varying float LightIntensity;
const vec4 ambient = vec4(0.13);
void main()
{
// check for shadowing. This func compare fragment against depth in shadow map. result is 0 or 1!
float isShadow = shadow2DProj(shadowMap, projShadow).r;
// fetch spotlight mask
float spot = texture2DProj(spotlight, projShadow).r;
gl_FragColor = (vec4(LightIntensity) + ambient) * spot * isShadow;
}
Dont forget to bind spotlight to some of tex units. Im not sure will shader work. It is code from a head right now, so it might be a few errors.
Well it’s a known shadow mapping problem. It’s happens only in situations where light source are very close to (infinite) triangle plane. To avoid this try to use bigger resolution of shadow map, smaller light frustum or shade such tris as dark as possible.