How to use the shader to determine whether the transparency of the pixel position is 0
If I understand you, the color you return from the fragmentshader is probably of the vec4 form
RGBA, meaning Read, Green, Blue and A-for-opacity(invers transparency).
If you let vec4.w=0.0 you get a fully transparent output. You set it to 1.0 for full opacity.
opengl-tutorial transparency