I was wondering how to properly structurize a set of nodes. In my game, sectors are loaded in the background while the players moves around. From what I know, it's best to keep the chunks as large as possible. While the HD is reading, the other threads can continue with the game. If that is true, it would be best to allocate the whole thing in one step, then read the data into the record-array. This time I need to find a way to store a large set of (pathfinding) nodes. Something like

<Delphi code>
TRelation = record
  targetNode : pointer; // pointer to another TNode
  data  : integer; // Some additional data stuff
  ...
end;

TNode  = record
  relations     : array of TRelation; // dynamic array
  relationCount : integer;  // count of related nodes
end;

nodes : array of TNode; // The array I'd like to write/read from a file

So, there is an array of nodes with a known side. But the relation count could differ for each node. Also loading and storing pointers doesn't work I guess, so I'll have to replace the "targetNode : pointer" with "targetNodeIndex : integer", which is an index in the "nodes" array. I could read node by node, and allocate the "relations" array for each node

   stream.read( nodeCount, 4 );   // read the count of nodes
   setLength( nodes, nodeCount ); // Allocate node array
   // Now start reading node by node
   for i:=0 to nodeCount-1 do begin
      stream.Read( nodes.relationCount, 4 );  // Read relation count into node
      setLength( nodes.relations, nodes.relationCount );
      // Read relations
      stream.read( nodes.relations, sizeOf(TRelation) * nodes.relationCount );
   end;

Works, but something tells me it can be done better. And when continously loading/unloading sectors, won't this fragmentate the memory? Greetings, Rick

20 MB...I whish:) All the nodes together might eat 1 gig. There are millions of them. However, I only load the sectors nearby the player, so probably I only need a few MB at a time. But due lots of loading/unloading, I was afraid of not doing it right.

The loading happens in the background in another thread. The loading doesn't have to be superfast. More important is not to block the game threads too much. From what I understand, other threads can just continue while reading. But in the code I gave, the reading is continuously interrupted with reading small bits and setting array lenghts. Maybe its not so bad, but if there is a better way...

But if I get your reply right, there's not really a better way to load? I'm not familiar with "Overlapping IO structures". I did a quick read on MSDN, but I don't think its really helping in this particular situation. But I could be wrong of course.

Look into hierarchical pathfinding. Doing any pathing on "millions" of nodes is going to be insanely slow, so you get two bonuses: you don't have to load the lower, finer-grained portions of the graph unless you absolutely need them for pathing; and pathing calculations are automatically faster because you reduce the size of the working set of nodes.

Well, there actually are two levels above this one for "global pathfinding". The millions of nodes will never be loaded or used at the same time. But when summing everything together, this is the estamated node count. One of the ideas was just to load all nodes initially, in order to save headaches with background loading, allocating memory, etc. But in this case the world is too big to load everything at once.