I think I've figured out what the subsectors do. At first I thought they formed triangles, but it seems they just form the border to a polygon. I was sure that after sorting the points into clockwise order that it would work.
Well it doesn't work. Appearently there are a bunch of sub sectors with only one segment in them. How they work I have no idea.
This looks like it has something to do with the missing triangles. Unfortunatly there are still some missing trinagles that don't match up with the last image, like the one where the player starts.
I'll probably have to implement the BSP tree, which I was hoping to avoid. [sad]