I'm trying to come up with a way of spatially describing objects in an environment. I've read a lot of articles and discussions on scene graphs and various methods of space-partitioning (Yes, I understand the difference between the two) but I still don't feel as though I any closer to deciding upon a good method.

I've got a very large game world, (3D, but mostly spread out horizontally and predominantly outdoors) which I'm going to be stream-loading content into. There's a large range of sizes that an object can be, everything from a tiny projectile up to huge buildings (Some smaller buildings which need to be easily added/removed from the world at times as well) and most objects are able to move in some way. I was considering using a quadtree/octree approach, but there are a few things putting me off that idea.

Firstly, if the game world is huge, it's presumably highly impractical to have a root node that's the size of the world. It would have to be split up many, many times until a node that encompasses just the currently loaded areas in created. One solution I suppose would be to divide the world up into a grid of potential large nodes, each one aware of it's neighbours, which can be dynamically grouped together to form the root node, or just be able to re-parent objects from one large node to another as they cross the boundaries. This seems overly complicated though, and makes it harder to find out simply where an object is. I'm also now of the impression that quadtree/octree designs aren't very good at tracking moving objects, I'm going to be rebuilding sections of the tree every frame when projectiles are in flight, for example. With a large number of moving objects, it seems inefficient to be constantly rebuilding the structure.

Part of the problem is that I'm not entirely sure what the responsibilities of the structure should be. Everything I read seems to describe a different set of problems being crammed into apparently similar designs. Using it to rapidly cull objects outside the field of view is an obvious use, is there a method that can easily incorporate occlusion culling as well? And if I'm representing the positions and rough sizes of objects in the world, should I expand it to assist in broad-phase collision detection as well?

Ok, well I've written a list of things which need to be handled in some way. What I need some help with is determining which of these can be easily handled together, and what should probably be separated into separate structures.

Frustum culling
Occlusion culling
Large terrain with LODs (viewed in the distance and up close)
Static objects (trees/buildings etc.)
Short-lived dynamic objects (particles / debris)
Long life-time dynamic objects (mostly characters)
Broad-phase to generate dynamic/static and dynamic/dynamic collision pairs
Light radii

Up to this point I've been organising the world in a simple uniform grid (where each cell is like a complete chunk of the world to be loaded), but that has a lot of limitations and it's starting to break down rapidly. It's quite a handy way of precisely locating objects in a very large world, but it doesn't play nicely with massive objects or objects that cross borders. Also, since each chunk was storing it's own terrain data, doing distant terrain is much much slower than it needs to be.

Ok, thanks, that's gives me a decent starting point. I'm still not entirely clear on the reasons for dividing up static and dynamic objects, is it just because a k-d tree can provide a better fit to the objects or something?