That is kind of odd. What is the distance of a chunk from? You might try bucketing the chunks a bit more by putting them in an octree or some other data structure. (These 16 chunks are in a leaf at dist X, who are in turn placed in branch that contains 4 other leafs, that has a distance of Y, etc)

Also, how often are you sorting? Sorting 15,000 things once shouldn't be a problem. Also, I have a sneaking suspicion that CompareTo probably offers terrible performance in Unity Mono. I would try doing that by manually comparing w values and seeing if you see any performance boost. EDIT: It all depends on whether it calls the CompareTo() that takes an object, or the concrete type. Some more internet searches seem to lean towards it calling the concrete type, and you should be fine.

EDIT2: Oh, but this seems to lean towards my suspicion, though that was for Unity on phone. Anyway, easy enough to check, and probably worth giving it a test. It also has a full listing of what CompareTo is doing for floats, and might explain any mild speedups you get with a more naive comparison. Which you might be able to get away with if you don't think you're going to get any NaNs or +/- Infinities in your list. (Which you could probably test for on insert, or elsewhere)

There is yet one more problem. The last laggy piece of code is

toLoad.Sort(delegate (Vector4 first, Vector4 second) { return first.w.CompareTo(second.w); });

Is there a faster way to sort objects? This one line is generating 50% of my lag.

I tried and failed to find the definition of "toLoad" in your previous code. Any information about the Vector4 class would also be helpful.

Why is this sorting just Vector4s? How are they linked back to the actual objects you need to sort? My C# is still a little rusty - is there something I'm missing here?

The Vector4s are the position and distance of a chunk. Any chunk can be loaded using its position. Vector4s are default structs in Unity.

That answers one question. What's the type of "toLoad"?

Edit: I see you have another thread where you explain more, so I'll participate there if I think of something to try. You don't need to answer me here.