Normally SwapBuffers synchronizes with the next vertical retrace (monitor frame). Disable sync-to-vblank to disable that behavior. Several ways. On Linux, “setenv __GL_SYNC_TO_VBLANK 0”, or in nvidia-settings, uncheck “Sync to VBlank” in OpenGL Settings. But the clean way (on all OSs) is to use the SwapInterval API. Search the archives for some good info there.
That won’t make SwapBuffers free (particularly if you have a complicated AA resolve that needs performed or a lot of GL work still in the queue), but it will stop it from busy waiting for the video scan-out. To minimize the amount of queued GL work processed inside of SwapBuffers, it can also be useful (for testing/timing use only!!!) sometimes to trigger a glFinish before SwapBuffers, to ensure that all the queued GL work is done before you enter SwapBuffers.
Use wglSwapIntervalEXT(0) or glXSwapIntervalSGI(0) to disable vertical synchronization. Note that this will introduce tearing and significantly reduce quality (some people cannot see tearing, others cannot stand it). If you must disable vsync, at least make it an option.
Also note that users can force-enable vsync through their drivers, regardless of what you try to do in your application.