'undefined' pixels in float texture fragmentshader

Hi all,

I come to this forum bearing a big problem in a fragment shader
that I’m working on - I’ve cornered it into this little piece
of code that gives me pixels that are neither a value, nor #inf
nor #nan, ( are they discarded ? ). my main function looks like:

1.vec4 flow = texture2D( VelDenMap, uv );
2.float p = flow.z;

3.float ad = smoothstep( 0.0, advection, clamp(p, 0.0, 1.0) )
4.gl_FragData[0] = vec4( ad );

advection is 0.8, VelDenMap is a 32-bit float texture
which is initialised to 0.0 ( im sure of that )
now, when I render the resulting float32-texture with a shader
that displays each pixel as

p = ( p - lower ) / ( higher - lower )

where lower and higher are the upper-and lower bounds for the range of values i want to display, everything outside of the
range [0,1] will be shown as a noise-texture, the image
remains black, even setting lower = higher = -10.0, or
lower = higher = 10.0.

however, when I change line 4 into

4.gl_FragData[0] = vec4( p );

all is fine, proving that p is a legal value. the same goes

4.gl_FragData[0] = vec4( advection );

also checking against NAN with

float inf1 = 1.0 / 0.0;
float inf2 = 1.0 / 0.0;
float nAn = inf1 * inf2;
if( ad == nAn )…
if( ad == inf1 )…

doesn’t detect ‘ad’ as being infinite or nan. ( thats why i
think the pixels may be discarded )

similar problems arise as soon as I use f.i. the ‘dot’ function.

does anyone have an idea what’s going on ?



I’m not sure that values can be “NAN” in GLSL, the GLSL spec states the result of some functions can be “undefined”.

Example: GLSL spec 1.20, page 57, section 8.2 Exponential Functions:

genType pow (genType x, genType y)

Returns x raised to the y power, i.e., x y
Results are undefined if x < 0.
Results are undefined if x = 0 and y <= 0.

I found this link once that illustrates the undefined behavior:


with this relevant section:
"A second pitfall to be aware of is that sometimes a variable may have an “undefined” value. Here is an example:

float x = pow(y, 2.0);
if (x >= 0.0 || x < 0.0) {
  // Set the fragment to be pure blue
  gl_FragColor = vec4(0.0, 0.0, 1.0, 1.0);
} else {
  // Set the fragment to be pure red
  gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);

You would expect the condition in the if statement to be true, and thus you would expect the fragment to be blue. However, if y is negative, the result of the call to pow() is undefined, so x is neither greater than, equal to, nor less than 0.0, so the fragment will actually be colored red.

Hi overlay,

Thanks for the reply, I’ve found this link too but it is
not what’s going on. all the values in my code seem valid
initially. it’s the moment that I use them in a function
like smoothstep, or dot that they become undefined, even
though the values are legal for that function…


by the way, if I change my textures to having GL_RGBA8 internal format rather than GL_RGBA32F_ARB, all is fine.

ok, it seems to have to do with the problem mentioned by
davidcornette after all; when i change line 3. to:

3.float ad = smoothstep( 0.0, advection + 0.00001,
clamp( p, 0.0, 1.0 ) );

all is fine. while the documentation to glsl states that the
result will be zero if x <= edge0, 1 if x >= edge1 and performs smooth Hermite interpolation between 0 and 1 when edge0 < x < edge1.

this does not seem to be the case, and the result seems to be in line with davids observations. Anyone have similar experiences ?


ok, solved. it’s the last line in the glsl language spec that gave me trouble:

smoothstep … Returns 0.0 if x <= edge0 and 1.0 if x >= edge1 and performs smooth Hermite interpolation between 0 and 1
when edge0 < x < edge1. This is useful in cases where
you would want a threshold function with a smooth
transition. This is equivalent to:
genType t;
t = clamp ((x – edge0) / (edge1 – edge0), 0, 1);
return t * t * (3 – 2 * t);
Results are undefined if edge0 >= edge1.

(pag. 58 )