Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 03 Jun 2003
Online Last Active Today, 09:19 PM

#5235870 IOCP shutdown design

Posted by on 20 June 2015 - 10:11 AM

I'll do another operations when send is completed rather than calling socket functions
on send completed, ill read packet from circular queue, format it and send again

That's a trivial amount of work for a modern CPU.

A loop that would be robust looks something like:

opcount = 1;
while (opcount > 0) {
    if (was_send) {
        opcount -= 1;
    if (time_to_close_socket()) {
    } else {
        if (read_was_completed) {
        if (data_to_be_written_and_no_write_pending) {
            opcount += 1;
If you use TCP, you could run this same loop for 30 sockets in the same thread loop and it would still be quite efficient.
If you're using UDP, you only need this one thread.

#5235780 IOCP shutdown design

Posted by on 19 June 2015 - 04:37 PM

have one request for wsarecv, and one more request for wsasend
should be able to abort socket without peer's disconnect
without any mutex, critical section or something(only atomic operations)

Then you need a reference count for number of requests outstanding.
When you want to close the socket, set a flag, then cancel the outstanding requests.
When the requests are cancelled, the flag tells them to not go back and re-queue. Instead, they reduce the reference count.
Whatever request reduces the count to zero, closes the socket.

That still has a race condition, though -- some thread may check the flag, see that it's clear, and then start preparing to call I/O.
Then the cancellation request comes in, but the OVERLAPPED isn't queued yet, so it doesn't get cancelled.
Then the thread actually issues the I/O, which will then actually wait for the I/O to complete.

The structure you want to use is not great threaded/asynchronous design.
How about you just use one thread, that does both WSASend() and WSARecv() and GetQueuedCompletionStatus()?
Then you don't have this problem. The amount of I/O done on the network card on a single socket will not be such that the single thread will be a problem.

#5235766 IOCP shutdown design

Posted by on 19 June 2015 - 03:34 PM

No, I don't have. I meant If I receive more than 10 packets in one second from peer (with single outstanding WSARecv), I want to close socket hard immediately.

My question is this:

Are you trying to receive from the same socket at the same time from more than one thread?
If so, how do you synchronize two threads getting messages from the same socket at the same time?

If not, if you only use one particular thread to get the queued completion status from a particular socket (or only have one request outstanding at a time,) then sequencing the code so that you don't issue a new asynchronous request when you decide to close the socket is easy.
Separately, if you have more than one outstanding asynchronous request, then you already have to solve this problem, because two of them could complete at the same time.

#5235722 IOCP shutdown design

Posted by on 19 June 2015 - 11:28 AM

If I have packet receive filter (e.g. 10 packets in one second) and peer exceeded it, I want to close socket hard and abort all operations from this user. So how can I do this?

If you receive multiple messages at the same time from the same remote end, how does your program synchronize that?
Synchronize this decision in the same way.

some libraries to imlement IOCP in painless way?

We are talking about asynchronous, multi-threaded programming here. There is nothing "painless" about it. That's why this is a hard area. It's totally doable (in the sense that computers are deterministic,) but it does require skill, determination, and measuring what you do.

That being said, I typically like the boost::asio library, because it has the same interface on multiple OS-es, and is reasonably efficient on each of them.

#5235707 IOCP shutdown design

Posted by on 19 June 2015 - 09:52 AM

1.Thread closes socket and 2. thread calling wsasend

This is a bug in your program.

If your program is structured such that this can happen, then your program has a bad connection management model. The only reason to call close() on a socket is if your program has determined that it has no more data to send or receive on the socket. Typically, this will be because the protocol has executed some "end of transmission" handshake.

In that case, the threads that do sending/receiving will detect this, and remove the socket from the pool of active sockets that get worked on, and then put it in a queue of sockets that should be closed.

#5234644 Common Internet Connection?

Posted by on 13 June 2015 - 01:48 PM

A good source might be the tech cert requirements for various consoles.
Not having done a console thing for the last few years, I don't know what they're up to now.
Note that you want to define some large part of the market as your target, which means you may be bottom trawling for the 90th, 95th, or 99th percentile.
256 kbps?
1 Mbps?

I'm getting a fairly consistent 3% packet loss over UDP on Comcast cable, using UDP ping to Google servers.
If the user connects via WiFi, that can easily add 50-100 milliseconds of jitter.
Cross-US-continent, you'll expect another 100 ms or so of latency.
Internationally, it can be as bad as you want.
Mobile, the equation changes -- with good signal, you can get fantastic throughput, but high latency.
Then someone else sits down next to the user at the cafe, and suddenly your throughput is cut by 10 and your packet loss doubles.

The best way to figure out what is right for YOUR game, is to define your actual market, and measure it as well as you can.
Euopre? Arabian peninsula? Australia? South America? International Space Station? Korea?
Also -- where do you host servers? Do users host servers? Then you can assume within-country. Do you host servers? Using Amazon, it's easy to get into at least three continents. If you can't pay the virtualization penalty, that gets a lot harder.

#5233844 Back-end server communication, Http or Manual TCP?

Posted by on 09 June 2015 - 11:42 AM

communication between servers that is not open to clients

The "standard" is typically to have all servers on a LAN (or VLAN) with a private address space -- 10.x for example.
Then, for clients to connect to servers, expose some number of IPs that are served by reverse NAT/proxies/load balancers.
A server talking to another server would then connect to a 10.x address, which is not reachable from the outside world.
A server can then know whether it's talking to another server, or to a client, by comparing the remote IP address -- if it's an IP address of a proxy, it's a client.
Don't mix up the IP address list used for proxies versus service servers :-) Best is to dedicate an entire subnet to the internal address of the proxies/DNATs. (All proxies live in 10.250.x.y, for example.)

If a client can somehow figure out how to talk to an arbitrary server (say, through a hole/config problem in your proxies) then you are vulnerable to internal server cross-call spoofing attacks.
Bonus points for using TLS with client and server authentication for internal communications.
Or, if you don't want to go through that effort, at least put a shared secret on each of the servers, and provide/verify this secret in a header/pre-amble of each connection.

#5232987 Client side hit detection vs server side with prediction

Posted by on 05 June 2015 - 12:24 PM

Isn't it shortsighted to invite cheaters when you know many of them exist?

Cheaters are not drawn to a game until there exists a large community so that the pay-off of seeming better than you are actually is worth the trouble.
To get to the point where you have a large community, you need to build a game that is fun to play.
Building a game that is fun to play is probably easier when using client-side detection and not worrying about cheaters yet.

It's better to have a popular game, and having to start coming up with anti-cheat code, than it is to have an unpopular game that dies because of a lack of players.
A popular game that doesn't have cheat problems is, of course, the best, if you have that option.

When considering schedule and technology, you really should consider the capabilities of your team and your budget, to avoid wasting effort of features you don't need /yet/.

#5232571 RTS Server architecture

Posted by on 03 June 2015 - 09:42 AM

You can't cheat if server says that player a hit points don't match server rules send less hit points and punish player a, and keep game running.

But you can cheat by reading out where the enemy units are, and what they are doing, even if you don't have line of sight to them.
In RTS-es, information/intelligence is important, so the incentive to cheat for those that need to "win" more than they need to "challenge themselves," is high.

#5232481 RTS Server architecture

Posted by on 02 June 2015 - 04:49 PM

We released an RTS game earlier this year that is client/server, with the server-authoritative state being updated to clients. It's not the most common approach, but it works

I think World at War did the same thing. The draw-back is that you simply cannot do 1,000 units per player in a game that's built that way. A 4v4 would be... laggalicious :-) With proper game design, the limited unit counts might not be a problem for a particular game, of course.

#5232397 Sending and Recieving Game Map Files.

Posted by on 02 June 2015 - 09:45 AM

But how do I connect to my lap top what do I need to run to connect.

Are you doing this for testing, or for external users to use?

Your laptop won't always be on, and its publicly visible IP is typically that of your current router/cable-modem/hotspot, not the local/private address on the network. For more about this, check the FAQ section on "how do I host a server for my game?"

For the testing case, if you're running a server on a port on your local machine, connecting to should work, from that same local machine.

#5232219 IOCP shutdown design

Posted by on 01 June 2015 - 01:42 PM

I just want to terminate listener socket and accepts because I'll recreate it. May be any risk-free implementation of kernel way?

What do you expect the accepting threads to do in this case?
How do they get told about the new socket?
Why do you want to re-create the listening socket? If it binds to the same port as a previously listening socket, you may need to use SO_REUSEADDR -- but also, what would be benefit be?

#5232169 MMORPG networking Solutions

Posted by on 01 June 2015 - 09:47 AM

You need to have an experienced network programmer working on this up-front (and ideally, also an experienced lead gameplay programmer), so they can design an architecture that's compatible with both your gameplay requirements and your wallet.


possibility to make the game run well as in free roaming the players can see them selfs and interact to other peoples to talk

Totally possible.

What's your budget for development time/resources?
What's your budget for operating the game?
Does each player actually pay a subscription cost, or is this "free to play" in any sense?

a single big server

The problem is that you can't get a "big enough" server because of physics. The necessary interconnect within the CPU chips scales by N-squared (or sometimes, exponentially,) which means that there is an upper limit to how "big" a CPU can be made, how "big" a RAM bus can be made, how "big" an I/O subsystem can be made, etc. And, as it turns out, even within the realm of the physically possible, getting a twice-as-big server will cost you at least four-times-as-much. And, pretty quickly when scaling up, "bigness" will start coming as multi-CPU servers with NUMA memory, and at that point, you have to develop for them just like you would develop for a set of separate servers connected with a very fast network. So, economies of scale say that it's better to have lots of medium-size servers (where each is a single- or dual-CPU system with 8-16 cores each, 32-128 GB of RAM, 10 Gb network interface.)

That being said -- 2000 players that walk around and chat, not shooting at each other? Totally doable on a single server instance.
The big challenge there will be what your client software will do when it gets told that there are 2000 separate players all in the same city square. Can you graphics engine handle that? Is there enough bandwidth to download all the avatar customization for each user?

#5232167 Sending and Recieving Game Map Files.

Posted by on 01 June 2015 - 09:42 AM

1/2: Either of the two options can work, if you implement them right.
3/4: Either of the two options can work, if you implement them right.

To distribute files, how about spinning up a HTTP server that serves the files. You can use If-None-Match and Etag headers to only actually transfer the map if the client has the wrong version. The benefit of this is that HTTP services / hosts / proxies / CDNs are very easy and cheap to set up.

The second best existing file transfer protocol is Bittorrent. You can look for a BT library or open source client you could drop into your code.

If you want a truly "drop in, no configuration needed" solution for any player to host a new map, then putting the download in your own protocol might be worth it. If you can't use TCP, though, you're going to have to bascially re-implement the TCP protocol semantics on top of UDP, which is doable, but really annoying and unlikely to be worth it. If you don't learn from TCP, then you will either end up with slow downloads, or you will end up with over-congesting links, which in turn leads to network saturation and slow downloads.

#5232165 IOCP shutdown design

Posted by on 01 June 2015 - 09:36 AM

You either need some kind of interlock, or you trust the kernel to do the right thing.

The interlock way: First, tell each worker thread to stop working; wait for all the threads to complete; then close the socket.

The kernel way: Trying to do an operation on a closed/invalid handle will return an error. Detect this error and break out of the thread loop.

The second way is less elegant, because there is a small chance that the same handle value gets re-allocated by some other part of your process.

In typical server applications, you never actually need to close any handles, until you want to shut down the server, and the simplest way to shut down a server is by just calling exit()/ExitProcess() -- the kernel will close the files/sockets/other-handles for you.