Hey

I actually asked this question in December last year on stack overflow but haven't gotten any responses yet, now this isn't strictly game dev question but I'm hoping that its still acceptable In this forum (if not I understand)

Here's the original question, Its probably a bit long winded so I'l try to sum it up https://stackoverflow.com/questions/53902687/how-to-get-rid-of-tcp-ip-send-delay-in-interprocess-communication-between-a-java

I have 2 separate processes running on 1 machine/vm. These processes do a lot of inter-process communication between each other via TCP sends. when running on a windows machine, a dedicated linux machine or a Open VZ vm everything works perfectly. There is no delay on individual sends.

When I run my setup on a KVM vm each tcp send gets about 20 ms of delay on it which makes my application unusably slow. I have enabled tcp no delay on both sides, makes no difference.

I have tested this on multiple different hosting providers. Dedicated linux servers and openvz vm's never has this delay, KVM vm's always has it.

Any idea what causes this? is this perhaps just a fact of life when using KVM? or can it be improved. 

I assume it might be an issue of partial-virtualization(openvz) vs full-virtualization(KVM) but I haven't tested on any other type of VM so i cant be sure.

Yes, it sounds a lot like a problem with the virtualization layer. Perhaps KVM virtualizes the network adapter at a rate of something like two 100 Hz ticks, or 1 50 Hz tick, or somesuch?

Things you can do to isolate and perhaps work around the problem:

1. Try using AF_UNIX sockets. These are bound to a file name, instead of a port. Some chosen file name in /tmp would be fine. See if the delay goes away.

2. Look at alternative KVM networking configurations. For example, create a private bridge that you bind your local TCP sockets to.

3. See if you can enable virtio mode for your KVM guest. (This may be a host restriction, though)

4. Try using SOCK_DGRAM sockets, to see if the problem is with the TCP implementation, or the IP/network-interface implementation.

In general, for IPC intra-machine, you'll want to use AF_UNIX or start the two processes from the same parent, using socketpair() or pipe() to create the IPC channel. Using a known port with TCP is higher overhead, runs the risk of colliding with the port, runs the risk of being caught in a too-strict firewall, and runs the risk of other processes finding the same port and interfering through whatever means.

Thank you,

I'l try setting up a private bridge and I'll check out virtio mode to see if I can solve this issue without changing away from TCP. If that doesn't work I'l try using AF_UNIX. I did read a little about it but I didn't look into it much since I just assumed Disk I/O from writing and reading to a file would be causing too much latency for IPC (of course I was wrong, It doesn't actually write anything to disk. I should have researched better in the first place)

One question though. I have multiple threads on both processes (depending on the amount of user requests being processed at any given moment). A thread on the one process will typically maintain 1 tcp connection to a thread in the other process until the user request is done and then the tcp connection is closed and both threads die. So if I were to use unix sockets the most logical thing for me would be to create a new set of "files" for each new thread pair. That means i'l be constantly creating new short lived "files", would this be problematic in any way or is there very little overhead?.

Why wouldn't you use exactly the same mechanism you use for TCP/IP? AF_UNIX SOCK_STREAM sockets use a file name, AF_INET SOCK_STREAM sockets use an IP address and a port number. Both of them have you call accept() to accept incoming connections, and each incoming connection is separate from the other.

Also, new thread, and new connection, per request is not generally very efficient. It will work, and it's not the end of the world of terrible, but if the two processes are long-lived, they should just establish a single connection between them in the beginning, and then funnel requests and responses across that connection. This is, unless you're doing this because there will be M and N processes in the future, with some kind of round-robin or consistent hashing look-up -- then you have to choose between building a mesh, or making a new connection per request, and the trade-off is not as obviously in favor of the mesh at that point, it depends on a variety of factors.

O ok, i just assumed you could only have 1 connection per "file". Thank you for clarifying that. 

In my application a new thread on both processes for each user request is essential but I could definitely consider tunneling all data between thread pairs through a single connection created when both processes start up. Just not a concern at the moment.

Thank you for pointing me in the right direction!

Hi,

I finally got around to switching over to AF_UNIX sockets. It resolved the issue. On KVM VM's there is now 0 delay on sends.

As a bonus it also increases performance of my application by roughly 25% across the board regardless of type of VM.

Thank you!

Glad to hear it! Hope your project is doing well, too!