Presumably, with this OpenGL extension:
More on this here. This is probably the legacy tessellator stuff they’ve had in their hardware for a while, not the new DX11-style tessellator stuff. But hopefully an ATI guru will chime in here.
For high-level details of what’s coming for the OpenGL Shader Model 5 functionality, see this:
Skip to 1:06:04 for SM5 in general, or 1:10:20 for the tessellation stuff in particular. Hopefully EXT/ARB/core support (and the NVidia hardware) will be out before/at GDC next year. Right now, it’s ATI-only, so just use their extension.
If so, the way its done on ATI cards, will Nvidia follow suit and do it in the vertex shader? Or is this going to be done in some other shader.
Huh? In DX11, it’s new shaders: hull and domain, after vertex before geometry sandwiching a new fixed-function tessellator gizmo. Don’t think it’d make sense for the model to be different in OpenGL. See that video I pointed you to. NVidia is calling them the “Control” shader and the “Evaluation” shader, sandwiching the “Primitive Generator”.