Sign in to follow this  
dhall3000

Strange client/server interaction when sending large amounts of data via C#/TCP

Recommended Posts

[font="Calibri"][size="3"]Hello,[/size][/font]

[font="Calibri"][size="3"]I've been a professional developer for quite a few years but have minimal networking experience. All of my minimal networking experience is in .NET (C#) and that is what I’m using now. I read the entire Networking forum FAQ and couldn’t find anything that addresses my issue (afaik).[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]I am running on a Windows 7 64 bit machine and am writing a data cache service. The idea is the ‘server’ (i.e. the cache) will be a service running on the local machine allowing other processes on the same machine to hit it with memory requests. The response data will sometimes be large (100MB won’t be uncommon) and I need it to be fast.[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]I am aware that UDP is faster but, although I need it fast, I need the guarantees that come with TCP more and am willing to pay the speed penalty. Also in regards to speed issue, most of the use case will be where the server cache service is fetching data for processes that run on the same machine so I won’t really be going out over the network, just hitting the loopback device.[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]On to my specific problem…[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]I have written a very simple server / client app pair to test speeds of large data transfers (1 billion bytes for my test). In the server code (below) in method GetBigBuffer() the top line is:[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]//server code: 1 billion bytes[/size][/font]

[font="Calibri"][size="3"]int size = 1000000000;[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]In the client code at the top of the Main() method I have:[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]//client code: 1 billion bytes[/size][/font]

[font="Calibri"][size="3"]int maxBytes = 1000000000;[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]If I run the code this way, things work great. In the client code, the do/while loop completes with only 1 iteration grabbing all 1 billion bytes in a single read in a respectable time of ~2500 milliseconds. However, if all I make is a single change in the server code as follows:[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]// server code: 100 million bytes[/size][/font]

[font="Calibri"][size="3"]int size = 100000000;[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]And make no changes in the client code, I get much worse performance! What happens is the do/while loop takes hundreds of iterations (775 in my most recent test) and takes 10’s of seconds (~44 in my most recent test) to get all 100 million bytes.[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]Further, if I then make an additional change to the client code as follows:[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]//client code: 100 million bytes[/size][/font]

[font="Calibri"][size="3"]int maxBytes = 100000000;[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]I find a return to the good performance as in the first scenario; the client do/while loop completes with 1 iteration with the total time to read all 100 million bytes taking ~250 milliseconds (very close to one tenth the time for one tenth the amount of data).[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]So things seem to perform very well when the size of the server’s write buffer matches the size of the client’s read buffer. Does anyone have any clue why that would be? Below is the full code for both my client and server apps.[/size][/font]

[font="Calibri"][size="3"][/size][/font]

[font="Calibri"][size="3"]Thanks![/size][/font]

//server code
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.IO;

namespace Server
{

public class Program
{

public static void Main()
{

Console.WriteLine("getting a big buffer");
byte[] dataToSend = GetBigBuffer();

Console.WriteLine("waiting for a connection request");
TcpListener tcpListener = new TcpListener(IPAddress.Any, 3000);

tcpListener.Start();

TcpClient client = tcpListener.AcceptTcpClient();

Console.WriteLine("established a connection");

TcpClient tcpClient = client;

NetworkStream clientStream = tcpClient.GetStream();

//this small buffer is to simulate receiving a data request from a client app
byte[] message = new byte[4096];
ASCIIEncoding encoder = new ASCIIEncoding();

try
{

//this read call is to simulate receiving a data request from a client app
int bytesRead = clientStream.Read(message, 0, 4096);

if (bytesRead > 0)
{

//print the pretend client data request to the console
string strMessage = encoder.GetString(message, 0, bytesRead);
Console.WriteLine(strMessage);

//now simulate dumping a lot of data back to the client
clientStream.Write(dataToSend, 0, dataToSend.Length);
clientStream.Flush();

}

}
catch (Exception e)
{
Console.WriteLine(e.Message);
}

Console.WriteLine("hit <ENTER> to quit");

tcpClient.Close();

Console.ReadLine();

}

public static byte[] GetBigBuffer()
{

//1 billion bytes
int size = 1000000000;

byte[] buf = new byte[size];

//populate the big buffer with fake data
byte[] bytes = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };

for (int i = 0; i < size; i++)
{
buf[i] = bytes[i % bytes.Length];
}

return buf;

}

}

}



//client code
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Diagnostics;

namespace Client
{

public class Program
{

public static void Main()
{

//1 billion bytes
int maxBytes = 100000000;
byte[] bigBuffer = new byte[maxBytes];

NetworkStream clientStream = GetClientStream();

ASCIIEncoding encoder = new ASCIIEncoding();
byte[] buffer = null;

//simulate sending out a request for data
buffer = encoder.GetBytes("simulated data request");
clientStream.Write(buffer, 0, buffer.Length);

int bytesRead = 0;
int count = 0;
int totalBytesRead = 0;

Stopwatch sw = new Stopwatch();

sw.Start();

//grab the incoming data from the cache server
do
{
bytesRead = clientStream.Read(bigBuffer, 0, maxBytes);
totalBytesRead += bytesRead;
count++;
} while (bytesRead > 0);

sw.Stop();

Console.WriteLine("num bytes received: " + bigBuffer.Length.ToString());
Console.WriteLine("time to fetch data: " + sw.ElapsedMilliseconds.ToString());
Console.WriteLine("hit <ENTER> to quit");
Console.ReadLine();

clientStream.Close();
clientStream.Dispose();

}

public static NetworkStream GetClientStream()
{

TcpClient client = new TcpClient();
IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 3000);
client.Connect(serverEndPoint);
NetworkStream clientStream = client.GetStream();

return clientStream;

}

}

}



Share this post


Link to post
Share on other sites
Played around with the code. Looks like:

- the server buffer is 100 million bytes
- the client buffer is 1 billion bytes
- the client calls this "bytesRead = clientStream.Read(bigBuffer, 0, maxBytes);" so it is attempting to read 1 billion bytes, but the server only sent 100 million. Not sure what the effect of that is ... trying to read 900 million bytes of nothing?

In debug mode I changed the Server maxBytes to 10 and the Client maxBytes to 20, then put in a break point after the read loop. The extra 10 bytes after the read are all just zeros.

Share this post


Link to post
Share on other sites
[quote name='EJH' timestamp='1312492554' post='4844711']
Played around with the code. Looks like:

- the server buffer is 100 million bytes
- the client buffer is 1 billion bytes
- the client calls this "bytesRead = clientStream.Read(bigBuffer, 0, maxBytes);" so it is attempting to read 1 billion bytes, but the server only sent 100 million. Not sure what the effect of that is ... trying to read 900 million bytes of nothing?

In debug mode I changed the Server maxBytes to 10 and the Client maxBytes to 20, then put in a break point after the read loop. The extra 10 bytes after the read are all just zeros.
[/quote]


Thanks for taking a look EJH!

so in client code when i say "bytesRead = clientStream.Read(bigBuffer, 0, maxBytes);" the .Read call will attempt to read 1 billion bytes? It seems more intuitive to me that the .Read call would see there's only 100M bytes to read, read that many and call it quits.

My thinking for making such a large client buffer is to ensure it has enough room to read all the incoming data in to. Do you know of a better practice that can read all the data (wheter it's 100M or 1B) in the fastest manner possible?

Thanks again...

Share this post


Link to post
Share on other sites
[quote name='dhall3000' timestamp='1312494497' post='4844725']
My thinking for making such a large client buffer is to ensure it has enough room to read all the incoming data in to. Do you know of a better practice that can read all the data (wheter it's 100M or 1B) in the fastest manner possible?
[/quote]

When you deal with large amounts of data, you almost always want to stream it one piece at a time, where the size of a piece varies based on application.
However, using a piece size bigger than 1 MB is almost never the right choice.

Regarding the number of times through the loop, it may be that the Windows TCP implementation sees that the sender has a 1 GB buffer, and the receiver has a 1 GB buffer, and is using memory mapping tricks to transfer it all. However, when the sender and receiver mis-matches, it just falls back to the standard TCP mechanism, which may packetize into whatever the MTU of your local interface is.

To test a network program for real, you need to run it over a network, and you need to make sure that more than one client tries to talk to it at the same time.

Btw: what you're trying to do sounds an awful lot like what memcached already does. Have you tried using that? Or another key/value store like membase, or redis, or whatever?

Share this post


Link to post
Share on other sites
[quote name='hplus0603' timestamp='1312495191' post='4844732']
[quote name='dhall3000' timestamp='1312494497' post='4844725']
My thinking for making such a large client buffer is to ensure it has enough room to read all the incoming data in to. Do you know of a better practice that can read all the data (wheter it's 100M or 1B) in the fastest manner possible?
[/quote]

When you deal with large amounts of data, you almost always want to stream it one piece at a time, where the size of a piece varies based on application.
However, using a piece size bigger than 1 MB is almost never the right choice.

Regarding the number of times through the loop, it may be that the Windows TCP implementation sees that the sender has a 1 GB buffer, and the receiver has a 1 GB buffer, and is using memory mapping tricks to transfer it all. However, when the sender and receiver mis-matches, it just falls back to the standard TCP mechanism, which may packetize into whatever the MTU of your local interface is.

To test a network program for real, you need to run it over a network, and you need to make sure that more than one client tries to talk to it at the same time.

Btw: what you're trying to do sounds an awful lot like what memcached already does. Have you tried using that? Or another key/value store like membase, or redis, or whatever?
[/quote]


Thanks hplus, very helpful information! The memory mapping trick is kinda what my hunch was as to the explanation for the speed difference.

So I looked in to memcached/membase/redis etc and a product called Microsoft Velocity kept coming up. I and a friend have spent some time playing with it and it seems like it might be the best solution what with .NET integration. Do you happen to know enough about this product to recommend its use or not use? (and I don't mean for my specific purpose...just in general)

Share this post


Link to post
Share on other sites
[quote name='dhall3000' timestamp='1312908060' post='4846763']
So I looked in to memcached/membase/redis etc and a product called Microsoft Velocity kept coming up. I and a friend have spent some time playing with it and it seems like it might be the best solution what with .NET integration. Do you happen to know enough about this product to recommend its use or not use? (and I don't mean for my specific purpose...just in general)
[/quote]

I think it's mostly Microsoft trying to re-invent Memcache/Redis, similar to how SQL Server is Microsoft trying to re-invent Oracle. I think it's kind-of early in its life, which means that it may have bugs, or MS may decide to kill it -- or it may be the best thing ever! I haven't use it; I have used Redis and Memcached from Windows and those products have worked fine. If you think it works fine for your use case, and you're already "married" to the Microsoft eco-system, I don't know any reason why you shouldn't try it! Please let use know how it goes if you do :-)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this