Which is to be expected. You aren’t moving a “camera” (OpenGL doesn’t have a camera), you’re moving the objects:
This transforms the positions of the vertices by the given matrix. If the matrix includes a translation in the positive X direction, the vertices will be moved in the positive X direction.
The viewpoint is always at 0,0,0 looking along the positive Z axis (conventional projection matrices as generated by e.g. glm::perspective or glm::ortho flip the Z direction, so it’s along the negative Z axis in eye space). To render the scene from a different viewpoint, you transform everything in the scene by the inverse transformation (e.g. to move the viewpoint to the right, you move everything to the left).
The easiest way to model a camera is to treat it like an object then pass the inverse of its transformation matrix to the vertex shader. Note that the projection shouldn’t be inverted, i.e. you want e.g. glm::perspective * glm::inverse(m) where m consists of the rotations and translations applied to the camera.
The fixed-function pipeline didn’t offer the ability to invert an arbitrary matrix, so the usual approach there was to invert the individual transformations (e.g. glTranslate(-x,-y,-z), glRotate(-angle,…), or glScale(1/x,1/y,1/z)) and apply them in the reverse order: for any matrices A,B,C, (A·B·C)-1 = C-1·B-1·A-1.