View matrix/model matrix woes

I’ve got the following hierachy:

TRANSFORM (translation, rotation, scaling)
   - CAMERA (view matrix)
   - MESH (model matrix)

The transform object has functions such as translate() and rotate() to modify its position and orientation in the world. To this transform, two objects are attached: a camera and a mesh which represents a first person weapon. When the transform is translated or rotated, i want the camera and the mesh to translate and rotate accordingly.

The camera requires a view matrix and the mesh requires a model matrix. Both of these matrices have to be computed every frame from the transform’s translation and rotation. I got it working using the following code:

// view matrix
mat4 rotationMatrix = glm::mat4_cast( m_rotation );
mat4 translationMatrix = glm::translate( mat4(), m_translation );

m_viewMatrix = rotationMatrix*translationMatrix;

// model matrix
mat4 rotationMatrixInverse = inverse( rotationMatrix );
mat4 translationMatrixInverse = inverse( translationMatrix );

m_modelMatrix = translationMatrixInverse*rotationMatrixInverse;

As you can see, I have to invert the rotation and the translation before using it to compute the model matrix, so that mesh and camera align. If I use the inverted values to compute the view matrix instead (as it should be), the camera’s orientation is distorted (not inverted).

So my question is: what do I have to change to be able to use the actual translation and rotation to compute the model matrix and the inverted values for the view matrix?