You have this same problem in a non-IOCP server, too! The main thing that changes with IOCP is how you structure your threads (one worker thread per CPU,) and how you structure your application to know when to do what (basically, build a state machine.)

There are many structures that can make sense. The main trick is going to be building something that is thread-safe enough for multiple completion threads to run simultaneously, yet doesn't serialize all logic on a single lock.

One way to structure a server like this would be:

1. There's a global hash table from "socket handle" to "client object." Inserts/removes/look-ups in this table need to be thread safe, but once you have the client object, the hash table doesn't need to stay locked.

2. The client object has an "incoming buffer" array of bytes, with a "head,"  "received," and "parsed" pointers (for where it's attempting to receive to, what has been received already, as well as how far it's managed to parse.) The buffer is treated as a cyclic buffer. The client object also has an OVERLAPPED structure to use for receiving into this buffer. As long as there is space in the in buffer, there should always be an outstanding receive call trying to receive into the empty space of this buffer, so each time a receive is completed, you should start another one (if there is enough empty space left,) then start parsing whatever you have.

3. The client object has another OVERLAPPED structure, plus an "outgoing buffer" array of bytes, with a "should write," "currently attempting to write" and a "written completed" pointer. As long as there is any data to write, you should have an outstanding write request, so when one completes, if there's anything in the buffer, start a new one. If something is added to the outgoing buffer, check whether a request is outstanding, and if not, start one. Note that you have to manage the "request outstanding" state yourself (with a flag or something,) because otherwise you will race with the OS.

4. The operation of "put stuff into the incoming buffer" must use a lock that's specific to the object, so each object probably has a lock (a CRITICAL_SECTION for example.) "Deal with data that was received" doesn't necessarily need a lock, if you "deal with data" before you "queue another read request." If you queue the read request before parsing the data, there's some risk that that second request will complete on another thread before you are done dealing with the data, so in that case, you also need a lock for the "deal with the data" read case. You'll typically also want to put a length value (say, 2 bytes, for a max length of 65535 bytes per packet) into the outgoing buffer before the actual bytes of the message.

5. "deal with the data" typically looks something like: Do I have less than 2 bytes? Just wait for more. Else, interpret those 2 bytes as a packet length, without consuming them. Do I have that many additional bytes? If not, just wait for more. Else, call some function to deal with a "complete, received packet, of size X" and then advance the "data read (buffer free)" pointer, and then go back to check whether you have another ready packet.

6. The handler for dealing with a packet may decide to forward the packet to other clients. To do so, iterate through the table of all clients, and call the "enqueue data" on each of those clients. The iteration over the table of all clients will need locking that table; calling write on each client object will lock each of those objects; there is risk for deadlock here if you're not careful!

7. You may run into the case where a client doesn't receive data fast enough, and there is no more buffer space left. You have the option of dropping the message and writing code that deals with not all messages getting to all clients, or detecting that the client can't keep up, and kick the client from the server. Those are your only two options. Specifically, you can NOT wait for the client to drain, because this will block a handling thread for an unknown amount of time -- potentially forever if the client is a suspended process that still responds to TCP ACKs but doesn't open the receive window. I recommend reasonably-sized buffers (256 kB per player?) and kicking players that don't receive their data fast enough.

8. Your "every so often" periodic message can be handled in a few different ways. One is to have a timer in the IOCP list of handles, and when it expires, iterate through all clients. Another is to have a separate thread which sleeps, and when it's time, wakes up, and iterates through all clients. A third is to have each client have a variable for when they last had one of those messages, and each time you receive something from the client (in the receive handling function,) check whether the time is right, and generate the outgoing packet and update the variable.

As you can see, there are many, many, bits and pieces that need to go together "just so" to make a high-performance, correct, asynchronous networking server. You need to understand locking, threading, cyclic buffers, contention, TCP protocol details, timing, and a number of other concepts. If you can learn and apply all of these, you can build a beautiful, event-driven server that makes very good use of available machine resources. If you're new to all of these concepts at the same time, you will probably find that trying to climb 8 different walls at the same time will be quite a challenge.

Good luck on your project, and make sure to keep us up to date on what you learn!