for some of the implementations, glmapbuffer will map the GPU memory into process virtual address space, that will be in the pinned memory and those pages will be locked out and wont get swapped out to disk unless we unmap it. so is there a limitation of the maximum number of mapped buffers, so if it has used all the existing memory and want to map another resource, how does it handled?
glMapBuffer() isn’t mlock().
There are several types of “resources” involved here: video memory, (physical) system memory, virtual memory (physical memory plus swap) and address space.
Address space is per-process, so there’s no contention there. A process can do what it wants with its own address space.
Physical system memory is a shared resource, but mapping video memory doesn’t require it. Or at least, doesn’t require exclusive access to it, i.e. locking physical memory into a process’ address space.
Video memory isn’t system memory. Whereas system memory is a shared resource, a process with access to the video hardware will typically be allowed to use as much video memory as is available. And whether video memory is mapped or manipulated via glBufferSubData() is irrelevant to resource consumption. Once a process allocates a buffer object, the implementation has to retain whatever is stored there until the client deletes the buffer.
And if the implementation stores buffers in (virtual) system memory, it will be treated like any other allocation of system memory. There’s no reason for the memory to be locked, whether mapped or not. Mapping just means that some storage is accessible via a stable address in the process’ address space.
The behaviour of glMapBuffer is well-defined: glMapBuffer
GL_OUT_OF_MEMORY is generated when glMapBuffer is executed if the GL is unable to map the buffer object’s data store. This may occur for a variety of system-specific reasons, such as the absence of sufficient remaining virtual memory.
And also:
If an error is generated, glMapBuffer returns NULL, and glUnmapBuffer returns GL_FALSE.
That should be sufficient to answer your questions, but note in particular the use of the term “system-specific” in the first quote I give: OpenGL itself doesn’t define a limit, but where a limit may exist it will depend on the system itself, i.e a combination of the hardware and the driver.