Constant manipulators size.

Im trying to draw 3d manipulators using a 3d perspective projection matrix and keep the size of the manipulators constant regardless if the viewport is resized. Right now what I do is to calculate the distance from the camera to the manipulator and affect a factor to it so the manipulator stay the same size. However as soon as the window is resized the manipulator grow accordingly.

Is there a way that I can still keep a 3d perspective and keep the manipulator a constant size like Blender, Maya or Unity?