I'm trying to add the lighting effect to the model. However, I don't know how to calculate the normal vector (some models have bevel). Is there any method to calculate normal vector? And, are the x, y, z, of normal vector must be in between -1 to 1?
Yes, Normals are between -1 and 1 for each of the three components.
Can’t you get the normals added to the models? This is usually the best approach.
Failing that, how about a normal map tetxure?
If your model does not have normals, they can be calculated with the cross product of two vectors lying on a plane (as in any two sides of a triangle). The normal should then be normalized, because the result from the cross product will not in general have unit length. This can be done in the geometry shader, because it is the only shader stage that knows about the entire primitive (other than the tessellation shaders, but that’s not relevant here), i.e., it can access all three vertices of the current triangle.
Regardless of where normals are determined, things get complicated if you have a mixture of “smooth” junctions and “sharp” junctions. By smooth junctions, I mean your triangles are approximating a smoothly curved surface. By sharp junction, I mean where two triangles come together and there should be a corner between them. One way to handle this is to group together all the triangles that approximate a single smooth surface, and for each vertex, average the normals of all the triangles that join there. (Another way is to do as BionicBytes suggested, and use a normal texture map. But if you haven’t got one, then you need some other way, such as described here.)
If you have grouped your triangles by surface, then you can easily calculate the “poor man’s” averaged normals within the GPU’s geometry shader if you specify the “triangles_adjacency” input layout qualifier. That not only lets you access the three vertices of the current triangle, but it also lets you access the third vertex of each of the three triangles that share an edge of the current triangle. That’s not as good as doing this calculation on the CPU because the geometry shader does not have access to any (if any) other triangles that share a vertex but not an edge with the current triangle. In other words, you’re not going to be able to average all the triangles that meet at each vertex, but only three of them, to calculate the averaged normal.
An advantage of calculating normals in the geometry shader is that that is after the ModelViewMatrix has already transformed the triangle vertices to eye coordinates. In other words, the normals are calculated in eye coordinates, so there is no need to rotate them along with the vertices as would be required with normals passed to the GPU from the CPU.
For example, I have a triangle. The vectors are (1,1,1), (0,0,0), (2,0,0). How can do the cross product to get the normal vector?
Did you try googling “cross product?” Wikipedia has a good write-up. I’d suggest reading about how it works, what it does, as well as how to do it. Wikipedia describes all of that, and with nice pictures to help get the points across. You really need to understand these things if you intend to use them, and it’s beyond the scope of these forums to explain something like that, especially given that it’s already been done so well by others, such as at wikipedia.