I'm guessing, that this is in screen-space coordinates (so somewhere between 0,0 and 1280,1024 as an example):

               Vector2 mouse = new Vector2(Mouse.GetState().X, Mouse.GetState().Y);

And the player position is in world-space coordinates (may be even 1000000, 1000000 as an example):

                Vector2 direction = mouse - player.GetPosition();

So subtracting the two may end up with seemingly random results!

It may be a mistake on my side, but I guess GetPosition based on its name does not do any transforming...
So you have to either transform the mouse position from screen-space coordinates to world space (e.g.: by using your view matrix) or transform the player position to screen-space, but the first one seems more "appropriate" and more useful (you may use the world position of the mouse for other stuff too, like when hovering an enemy you could display its health-bar or something like that).

Plus, this is just a hint to help you write tidy code:

if(Mouse.GetState().LeftButton == ButtonState.Pressed)
{
int speed = 5;
Vector2 mouse = new Vector2(Mouse.GetState().X, Mouse.GetState().Y);

Try to call Mouse.GetState only once per frame, and use the fileds/properties of that cached instance in your code (button pressed state, X, Y etc...).
There is no real reason to get the state multiple times per frame, the state structure is rather big + it may call down to OS level to poll the sate (may not be exactly a cheap call).

Br.

It's called whenever the left mouse button is pressed.

Then why are you checking the button state? You're showing code that doesn't appear to contain a bug. If you want help to find the bug you need to provide more context. If you're like most of the people who swing through the FB forum you'll respond to this by telling me a very long story that only contains information that I don't need, then I'll get frustrated and leave. Please post all of the relevant code instead.

After testing it again it seems like the movement direction is the last direction the mouse moved and not towards the mouse.

Try printing the mouse and player coords to the screen every frame. Spidi is probably on the right trail, since it would generate that kind of behavior; if the player is at the indicated location and then the mouse moves then the direction to the new indicated location will be the direction that the mouse moved.

I've checked your camera code, I even created a small XNA project to try it out, because it seemed to be correct (haven't found any problems in it).
And indeed, it works correctly, even with zooming and rotation.

Now two options seem logical.
1.: you either have some problems regarding code you have not checked/shared yet (e.g.: drawing logic, camera control).
2.: you actually do have a problem with your mouse handling code you originally shared and the problem is still what I've pointed out originally, which is the missing/incorrect world space-transformation of your mouse coordinates.

At current stage, I can not help with case #1, since I don't know what else you do with this piece of camera code when drawing or when changing its position/rotation/zoom...
But if case #2 is the real problem I can still nudge you in the right direction! Here is the code I used in the project with your camera code:

// in the Game class implementation:
Vector2 player = Vector2.Zero;

protected override void Update(GameTime gameTime)
{
	var dt = (float)gameTime.ElapsedGameTime.TotalSeconds;
	
	// controlling camera position/rotation/zoom...
	
	// move the player towards the mouse:
	var mouse = Mouse.GetState();
	if (mouse.LeftButton == ButtonState.Pressed)
	{
		// take the mouse "screen-space" position and transform it to "world-space"
		var mouseScreenSpace = new Vector2(mouse.X, mouse.Y);
		// inverse matrix transform, because the view-matrix transforms from "world-space" to "screen-space",
		// the mouse position is in "screen-space", so we need the inverse to get its "world-space" origin...
		var inverseView = Matrix.Invert(camera.GetViewMatrix());
		var mouseWorldSpace = Vector2.Transform(mouseScreenSpace, inverseView);
		
		// take the direction of the player towards the "world-space" position of the mouse
		var direction = mouseWorldSpace - player;
		var length = direction.Length();
		// use delta time in seconds since last frame for determining the movement speed (and for having smooth movement)
		var speed = 250f * dt;
		if (length > speed)
		{
			// apply the calculated speed to the player position
			direction.Normalize();
			player += direction * speed;
		}
		else
		{
			// the player is already really close to the camera
			// simply set its position to camera "world-space" position
			// otherwise its position would jitter near the mouse...
			player = mouseWorldSpace;
		}
	}
}

As I mentioned, this piece of code worked perfectly with your Camera2D class in an empty project, even with camera position/rotation and zoom changes!
Hope it helps!