IOCP shutdown design

Networking and Multiplayer Programming

Started by rigel May 31, 2015 05:46 PM

17 comments, last by Taylor Softs 8 years, 9 months ago

11,916

June 19, 2015 09:34 PM

No, I don't have. I meant If I receive more than 10 packets in one second from peer (with single outstanding WSARecv), I want to close socket hard immediately.

My question is this:

Are you trying to receive from the same socket at the same time from more than one thread?
If so, how do you synchronize two threads getting messages from the same socket at the same time?

If not, if you only use one particular thread to get the queued completion status from a particular socket (or only have one request outstanding at a time,) then sequencing the code so that you don't issue a new asynchronous request when you decide to close the socket is easy.
Separately, if you have more than one outstanding asynchronous request, then you already have to solve this problem, because two of them could complete at the same time.

enum Bool { True, False, FileNotFound };

rigel

401

Author

June 19, 2015 10:18 PM

Are you trying to receive from the same socket at the same time from more than one thread?

sorry for my dumbness, i cant adapt to conversation due to my english weakness

its better if you write pseudo codes or any source codes to look out

actually what I want do is,

create two threads per cpu for iocp

have one request for wsarecv, and one more request for wsasend

should be able to abort socket without peer's disconnect

without any mutex, critical section or something(only atomic operations)

hplus0603

11,916

June 19, 2015 10:37 PM

have one request for wsarecv, and one more request for wsasend
should be able to abort socket without peer's disconnect
without any mutex, critical section or something(only atomic operations)

Then you need a reference count for number of requests outstanding.
When you want to close the socket, set a flag, then cancel the outstanding requests.
When the requests are cancelled, the flag tells them to not go back and re-queue. Instead, they reduce the reference count.
Whatever request reduces the count to zero, closes the socket.

That still has a race condition, though -- some thread may check the flag, see that it's clear, and then start preparing to call I/O.
Then the cancellation request comes in, but the OVERLAPPED isn't queued yet, so it doesn't get cancelled.
Then the thread actually issues the I/O, which will then actually wait for the I/O to complete.

The structure you want to use is not great threaded/asynchronous design.
How about you just use one thread, that does both WSASend() and WSARecv() and GetQueuedCompletionStatus()?
Then you don't have this problem. The amount of I/O done on the network card on a single socket will not be such that the single thread will be a problem.

enum Bool { True, False, FileNotFound };

rigel

401

Author

June 19, 2015 11:14 PM

Then you need a reference count for number of requests outstanding.
When you want to close the socket, set a flag, then cancel the outstanding requests.

If I understand right, you mean wait operations to be completed and don't post it again. When all is completed, close socket.

If so, I don't care operations for complete gracefully because its abortive close.

I don't want to call closesocket function simultaneously with any wsa operation(but any operation should be called simultaneously except closesocket, maybe socket handle going to reused), so I want to do something like this:

WSABegin() // will check for socket is closing, if so reads ref count if zero, closes socket here and sets invalid handle else adds ref

WSASend(), WSARecv(), ... any socket required operation

WSAEnd() // decrements ref and checks for ref is zero, if so checks for closing flag, sets state closed and calls closesocket

void close()

{

InterlockedExc(&m_state, closing)

if (m_ref == 0)

{

closesocket(socket)

socket = INVALID_SOCKET

}

some complex to explain for me

not sure how to implement these operations in atomic way

is that way good?

The structure you want to use is not great threaded/asynchronous design.

sounds like I dont know what im doing actually

How about you just use one thread, that does both WSASend() and WSARecv() and GetQueuedCompletionStatus()?
Then you don't have this problem. The amount of I/O done on the network card on a single socket will not be such that the single thread will be a problem.

I'll do another operations when send is completed rather than calling socket functions

on send completed, ill read packet from circular queue, format it and send again

hplus0603

11,916

June 20, 2015 04:11 PM

I'll do another operations when send is completed rather than calling socket functions
on send completed, ill read packet from circular queue, format it and send again

That's a trivial amount of work for a modern CPU.

A loop that would be robust looks something like:


opcount = 1;
WSARead(OVERLAPPED);
while (opcount > 0) {
    GetQueuedCompletionStatus();
    do_data_processing();
    if (was_send) {
        opcount -= 1;
    }
    if (time_to_close_socket()) {
        CancelIoEx();
    } else {
        if (read_was_completed) {
            WSARead(OVERLAPPED);
        }
        if (data_to_be_written_and_no_write_pending) {
            WSASend(OVERLAPPED);
            opcount += 1;
        }
    }
}
closesocket();

If you use TCP, you could run this same loop for 30 sockets in the same thread loop and it would still be quite efficient.
If you're using UDP, you only need this one thread.

enum Bool { True, False, FileNotFound };

rigel

401

Author

June 20, 2015 08:22 PM

CancelIoEx functions sounds good for me nevertheless, it has been rarely used in IOCP examples, why? any performance issues?

hplus0603

11,916

June 20, 2015 09:51 PM

CancelIoEx functions sounds good for me nevertheless, it has been rarely used in IOCP examples, why? any performance issues?

No, the problem is the race condition I described in my previous post. It is only useful if you can guarantee that no more requests will be enqueued on the file descriptor, which in turn means that you need to hold a lock when creating new requests, as well as holding a lock when cancelling.

Why do you not want to hold locks? A CRITICAL_SECTION is generally as cheap as an interlocked atomic operation, unless you actually have contention.

Also, why do you want to close the socket on too many messages?
If the remote client has a bug, then you should be able to send a "please shutdown" message to the other side, and it should shut down.
If the remote client is malicious, it's not like it will stop sending packets on the network to you just because you close the socket.

enum Bool { True, False, FileNotFound };

rigel

401

Author

June 21, 2015 08:41 AM

Why do you not want to hold locks? A CRITICAL_SECTION is generally as cheap as an interlocked atomic operation, unless you actually have contention.

I believe its possible without locks, it should do other operations rather than being locked?

Also, why do you want to close the socket on too many messages?

If the remote client is malicious, it's not like it will stop sending packets on the network to you just because you close the socket.

true but maybe remote client is flooding some packet that does massive operations and lag server

Taylor Softs

146

July 06, 2015 01:55 AM

sorry for my english

you need to understand something about multi-threading to handle multiple socket at the same time

CRITICAL_SECTION is needed in your case to synchronous your thread or you will get Access violation when you try to access a handle of a closed socket

all right but the problem is not solved

you need to add some boolean variable called bShutDownServer like this

EnterCriticalSection(&g_Cr);

closesocket(yoursocket);

bShutDownServer = TRUE; // if you are not in MFC just use true

LeaveCriticalSection(&g_Cr);

in the loop of your handler accept

you need to do the same thing

EnterCriticalSection(&g_Cr);

//safe thread here

if (!bShutDownServer)

AcceptEx(...);

else

{

//Set your socket to INVALID_SOCKET

// cleans everything to avoid memory leaks and call your destructor class after you leave the critical section

}

LeaveCriticalSection(&g_Cr);

PS : i use SlimReadWriter(SRW) instead of CRITICAL_SECTION because the first one is more fast in the kernel level synchronisation (Source : Book Windows Via c++ Jeffrey richter)

IOCP shutdown design

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

IOCP shutdown design

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines