In the official reference the description for glViewport() is

glViewport specifies the affine transformation of x and y from normalized device coordinates to window coordinates. Let (xnd, ynd) be normalized device coordinates. Then the window coordinates (xw, yw) are computed as follows:

xw = (xnd + 1) · (width / 2) + x
yw = (ynd + 1) · (height / 2) + y

My question is: what does this “normalized device coordinates” mean?

After the perspective matrix the coordinates are in normalized format. Normalized means that they are in the range from -1…1 (with 0,0 being the center of the viewport)

Thanks. Is there a function to get the normalized device coordinate from a 3D coordinate directly? Or is there a function to get the normalized device coordinate from the screen coordinate?

There is a nice diagram in the opengl spec, section 2.10 “Coordinate transformations”, figure 2.6 (on page 32). The normalized coordinates are obtained after the perspective division is applied to the clip coordinates.
Why the people don’t bother to read the spec!?
There are no funcs of the kind you ask of in opengl itself - it does the transforms internally, for itself only. In the glu supplementary library there are some such helper funcs, or you can just do the transforms you wish by yourself.