1. I've been able to build smaller test sets (~20 * ~20) manually like this:

s = start
f = goal
1 = passable
0 = impassable

f, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1,
1, 0, s, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1,
1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1,
1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1,
1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1,
1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1,
1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1,
1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1,
1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1,
1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1

But constructing large sets (like 1000 * 1000) by hand is not feasible. Are there any tools or available test files available one can use to test ones implementation of a* alghorithm?

2. Reading the pseudo-code on wiki:

function A*(start, goal)
    // The set of nodes already evaluated.
    closedSet := {}
    // The set of currently discovered nodes still to be evaluated.
    // Initially, only the start node is known.
    openSet := {start}
    // For each node, which node it can most efficiently be reached from.
    // If a node can be reached from many nodes, cameFrom will eventually contain the
    // most efficient previous step.
    cameFrom := the empty map

    // For each node, the cost of getting from the start node to that node.
    gScore := map with default value of Infinity
    // The cost of going from start to start is zero.
    gScore[start] := 0 
    // For each node, the total cost of getting from the start node to the goal
    // by passing by that node. That value is partly known, partly heuristic.
    fScore := map with default value of Infinity
    // For the first node, that value is completely heuristic.
    fScore[start] := heuristic_cost_estimate(start, goal)

    while openSet is not empty
        current := the node in openSet having the lowest fScore[] value
        if current = goal
            return reconstruct_path(cameFrom, goal)

        openSet.Remove(current)
        closedSet.Add(current)
        for each neighbor of current
            if neighbor in closedSet
                continue		// Ignore the neighbor which is already evaluated.
            // The distance from start to goal passing through current and the neighbor.
            tentative_gScore := gScore[current] + dist_between(current, neighbor)
            if neighbor not in openSet	// Discover a new node
                openSet.Add(neighbor)
            else if tentative_gScore >= gScore[neighbor]
                continue		// This is not a better path.

            // This path is the best until now. Record it!
            cameFrom[neighbor] := current
            gScore[neighbor] := tentative_gScore
            fScore[neighbor] := gScore[neighbor] + heuristic_cost_estimate(neighbor, goal)

    return failure

function reconstruct_path(cameFrom, current)
    total_path := [current]
    while current in cameFrom.Keys:
        current := cameFrom[current]
        total_path.append(current)
    return total_path

In my case, as in the layout I showed in #1, the distance between a node and its neighbours are always 1 (adjacent). The "gScore" always increases by one each step away from start node.

Dosn't that mean that mean I can simplify it so whenever we add a node we have the optimal path between the start node and it?

So either the node is in the open or closed set and we ignore it or its not visited and we add it, like below:

function A*(start, goal)
        ...
        for each neighbor of current
            if neighbor in closedSet or openSet
                continue
            
            openSet.Add(neighbor)
            cameFrom[neighbor] := current
            gScore[neighbor] := gScore[current] + 1
            fScore[neighbor] := gScore[neighbor] + heuristic_cost_estimate(neighbor, goal)
        ...

.....@....#.............#.........................
..........#.............##........................
..........#..............#........................
..........#.......................................
.....#################..#########.###########.....
.....................#...............#......##....
.....................#...............#....*..#....
.....................................#.......##...
.....................#...............#...#....#...
.....................#.......#.......#...#........
.....................#.......#.......#...#....#...

Yeah, for my purposes, doing as Norman suggests is all I did. Random chance of blocking a tile. It's by no means perfect, but it worked for me. If you're a more corridor style game, you may want to look into generating maps that way to test your pathfinding. Though if you're doing A* that part doesn't matter, unless you start adding cooperative components or want to deal with blocking/obstacle avoidance, etc, and then you'd need it.

Thanks for input. I found very large test maps. It's at http://movingai.com/benchmarks/dao/ if anyone else wants to know.