Here’s an excerpt from The OpenGL Extensions Guide.
Environment-mapped bump mapping is performed by calculating the reflection of the direction to the camera at each fragment and using the resulting vector to fetch a sample from a cube texture map containing an image of the environment. This environment cube map is typically stored in world space so that it can be generated without any dependency on a particular model’s local coordinate system.
The reflection vector calculation depends on the normal vector fetched from the bump map at each fragment. Since samples from the bump map are stored in tangent space, we must transform them into world space before the texture shader can determine the appropriate reflection vector. This is accomplished by calculating a 3x3 transformation matrix at each vertex that maps tangent space to world space and storing its rows in three sets of texture coordinates. Three dot products are evaluated in the texture shader stages to perform the matrix-vector multiply needed to transform the normal vector into world space for each fragment.
Let M be the 3x3 matrix that transforms 3D vectors from model space to world space. Typically, the normal vector N, the tangent vector T, and the bitangent vector B (where B = ±N x T) are available in model space at each vertex. The matrix W that transforms 3D vectors from tangent space to world space is given by
...........[ Tx Bx Nx ]
W = M [ Ty By Ny ].
...........[ Tz Bz Nz ]
The rows of the matrix W are stored in the s, t, and r texture coordinates for texture units 1, 2, and 3 for each vertex. As shown in Table 6.3.8, dot products are performed in these texture shader stages in order to transform the normal vector fetched in stage 0 into world space.
When reflecting the direction to camera E (the eye vector) across the world-space normal vector at each fragment, we may choose to use a direction to camera calculated at each vertex or a constant direction to camera. The GL_DOT_PRODUCT_REFLECT_CUBE_MAP_NV texture shader operation assembles the x, y, and z coordinates of the 3D eye vector from the q texture coordinates corresponding to texture units 1, 2, and 3, respectively. These components of the eye vector must be calculated in world space at each vertex and are obtained by evaluating the equation
E = M (C - V),
where C is the model-space camera position and V is the model-space vertex position. The vector E does not need to be normalized since its reflection is used to sample a cube texture map, for which vector length is irrelevant. As an alternative to calculating the eye vector at each vertex, a constant eye vector may be used by specifying the texture shader operation GL_DOT_PRODUCT_CONST_EYE_REFLECT_CUBE_MAP_NV for stage 3. In this case, the eye vector E is given by the value of GL_CONST_EYE_NV for texture unit 3.
The world-space normal vector calculated in stages 1, 2, and 3 may optionally be used to fetch a sample from a diffuse cube texture map in stage 2. If the texture shader operation for stage 2 is GL_DOT_PRODUCT_DIFFUSE_CUBE_MAP_NV, then the transformed normal vector is used to access the cube map bound to texture unit 2. The result of the cube map fetches in stages 2 and 3 can be added in the texture environment or in register combiners. If no diffuse cube map fetch is desired, then the texture shader operation for stage 2 should be set to GL_DOT_PRODUCT_NV.
Table 6.3.8 Texture maps, texture shader operations, previous texture inputs, and texture coordinate values used to perform environment-mapped bump mapping. Stage 2 differs in the two cases that (a) a diffuse lighting cube map is used, and (b) only the environment cube map is used. Stage 3 differs in the two cases that (1) the eye vector is calculated at each vertex, and (2) the eye vector is constant. The matrix W is the 3x3 matrix that transforms normal vectors from tangent space to world space, and the vector E is the world-space direction to the camera.
Stage / Texture map
0 Bump map
2 (a) Diffuse lighting cube map
… (b) None
3 Environment cube map
Stage / Shader Operation
2 (a) GL_DOT_PRODUCT_DIFFUSE_CUBE_MAP_NV
… (b) GL_DOT_PRODUCT_NV
3 (1) GL_DOT_PRODUCT_REFLECT_CUBE_MAP_NV
… (2) GL_DOT_PRODUCT_CONST_EYE_REFLECT_CUBE_MAP_NV
Stage / Previous texture input
Stage / Texture coordinates
0 Ordinary 2D texture coordinates
1 (s,t,r) = W1 ; q = Ex
2 (s,t,r) = W2 ; q = Ey
3 (s,t,r) = W3 ; q = Ez
[This message has been edited by Eric Lengyel (edited 01-03-2004).]