when zoomed in and I touch both fingers it causes a jump.

How do you track motion while zooming? Probably by the centre between two fingers?

Anyway, when you just found second touch (zoom/drag started), don't apply motion to objects on that event.

Only save initial position of anchor point. On next motion events you'll calc difference between new anchor position and initial position, and move your object proportionally to that difference vector.

What 'space' should I work in?

Doesn't really matter. You'll scale motion vector anyway, only coefficients will change.

How do I store my values?

I find it convenient to store location/look_at points and up vector. Normalized direction vector and right vector recalculated on location/look_at changes. It's trivial to move camera then.

When zooming I found I needed to have an extra zoom variable that was temporarily applied while using two fingers then permanently applied when I released (this made things messy).

No need in separate zoom variable. On first touch of second finger you'll save initial distance between touches.

On next motion events you'll find new distance. Relative difference in distance can be treated as motion of "zoom dial", which is directly applied/converted to camera's scale/fov/whatever.

Translation need to go slower when zoomed and faster when zoomed out.

Since you're moving it in plane parallel to screen, you'll need cam's up/right vectors, nonlinearly scaled it proportionally to current zoom (that depends on your projection), scaled by touch motion vector and added to camera position.

Then when I drag to another point Bscreen it should still map to Aworld which I found previously now I have:

Ah, I missed the point.

Ok, considering the case when you store camera's location and scale parameters, and not final matrix.

To scale something around selected centre, you move that centre to origin, scale around origin, then move back to selected centre.

Transform matrix should look like this:

[attachment=30155:scale_around_centre.gif]

Or in its final form:

[attachment=30156:scale_final.gif]

s is scale factor, x/y is scaling centre.

Now, you have screen coordinates of a scaling touch, it needs to be unprojected into world space to get scaling centre x/y. You build that matrix, and transform camera's location with it.

After setting camera to new location and new scale, entire scene will move/scale, while maintaining screen location of a scaling centre.

Here's sample code in Lua. Camera stores its location in world space and zoom factor (inverse of scene geometry scale).

local function scale_around_centre(point, scale, scaling_centre)
    local sm = {
        {scale, 0, scaling_centre[1]-scale*scaling_centre[1]},
        {0, scale, scaling_centre[2]-scale*scaling_centre[2]},
        {0,     0, 1}
    }

    local x,y = point[1], point[2]
    return {
        x*sm[1][1]+y*sm[1][2]+sm[1][3],
        x*sm[2][1]+y*sm[2][2]+sm[2][3]
    }
end

-- current state of camera
local camera = {
    location = {0,0}, -- camera location in world space
    zoom = 2          -- zoom >1 means zoom out
}

local new_zoom = 6 -- desired new zoom
local screen_touch = {0.75,0.25} -- dummy touch in screen space

-- unproject touch
local world_touch = {
    screen_touch[1]*camera.zoom+camera.location[1],
    screen_touch[2]*camera.zoom+camera.location[2]
}

local relative_scale = new_zoom/camera.zoom
local new_location = scale_around_centre(camera.location, relative_scale, world_touch)

-- set camera new parameters
camera.location = new_location
camera.zoom = new_zoom

-- calc new screen location of same world_touch
local new_touch = {
    (world_touch[1]-camera.location[1])/camera.zoom,
    (world_touch[2]-camera.location[2])/camera.zoom
}

-- new_touch should be equal to original screen_touch

Was writing it too long

Check if this math simplifies something.

what do you use to generate those matrix images?

It's http://www.wolframalpha.com