Polygon trianglize arithmetic?

I encount concave polygon and polygon with holes. I want to use a serial triangles to represent such model.

Any souce code available about such topics?

Look into glu tesselation functions. They let you specify a contour and holes in the polygon, and tesselate it into triangles through a callback functions mechanism.