Crazy flickering with simple zoom function

I have a 2D setup, with a quad with a texture. I want to be able to zoom in and out on it, I have come up with this, but it semi-works.

This is in the renderer

            Dim xy As New Vector2(Mouse.X / Width * 5000, Mouse.Y / Height * 5000)
            GL.Translate(xy.X, xy.Y, 0)
            GL.Scale(1.5 ^ ZoomFactor, 1.5 ^ ZoomFactor, 0)
            GL.Translate(-xy.X, -xy.Y, 0)

Zoom factor changes with the scroll wheel

This works kinda, but as I move the mouse around generally, the translate changes by little bits and causes the quad to move with the mouse.

If I have it so that xy is set in the scroll wheel function it causes this problem:

If I keep my mouse in one spot, it zooms to that point fine, but if I zoom in a bit, then move the mouse and zoom, it makes huge jumps to somewhere else completely. I need to stop that, I think it has something to do with my unprojecting code.

(I changed the post, as I fixed my original problem, but this one popped up instead)