Sign in to follow this  
catch

Small file transfers?

Recommended Posts

catch    192
Anyone know of a very small and/or lightweight API/lib for doing file transfers? Basically, need to do a few small file transfers (100k ish) and I'm uncertain of a decent starting point. I want to send map information to a client from the server... packets might work, but it would take a bunch... O.o

Share this post


Link to post
Share on other sites
Evil Steve    2017
What protocol? TCP? UDP?

You'll probably be quickest using whatever you're using now for a relatively small file - If you're using UDP, some sort of basic reliable packet system would work quite well.

Share this post


Link to post
Share on other sites
catch    192
TCP.

I'm using a pretty basic, but robust system, which is a net event system (XML marshaling).


I mean, 100k isn't much at the end of the day. I'm having trouble deciding if I want to send global maps everytime a player logs in, or if I just want to send the map data as they switch maps.

It would be easier just to drop a map file into the map directory for the client. It might be more footwork to send a couple map packets with map data in it, at least, it would be kind of wasteful, data-transmission wise.

I don't know. I figured there's got to be some small module out there that just does basic TCP file-xfers?

Cross platform preferred, but the primary system would be win32.

Share this post


Link to post
Share on other sites
SuperRad    163
I was going to suggest this jokingly but if it works for you why not.
You could bundle wget (windows build) with your client, stick the map files on a webserver, and run a system command to pull down those files to the map folder.
Sure it requires calling an external program, but wget is fairly robust and this will require minimal work from you.
[edit] You could even use ftp thats supplied with most windows versions(short of 3.11 and below) to pull the files down from anonymous ftp ftp scripts[/edit]

Share this post


Link to post
Share on other sites
hplus0603    11356
If the file transfer is from some server you know, then use HTTP, and use whatever HTTP client is convenient for your language. In C/C++, there's for example libcurl, or the ultra-small HTTP-GET.

If the file transfer needs to be fancy, like doing peer-to-peer or from untrusted sources with digital signatures, then you'd have to give us a little more information for us to answer the question. It's very unlikely that rolling your own, custom protocol on top of TCP is really the right answer, except in certain narrow cases.

Share this post


Link to post
Share on other sites
catch    192
Yeah, it would be a trusted source (server). And nothing fancy required.

Basically it's just downloading the most current map versions so the client has the most up to date map the server has.

Then again the map data is quite simple, at this point I'm wondering if just streaming it as packets might be feasible. Then again, if texture/tile data doesn't change regularly, why waste the bandwidth... We're talking maybe 20 tiles (3 ints for data) in the buffer area...

Share this post


Link to post
Share on other sites
wood_brian    193
Quote:
Original post by catch
TCP.

I'm using a pretty basic, but robust system, which is a net event system (XML marshaling).



I have an API that transfers files. I'm not sure how it would work with XML stuff. I've never tried it with that. There's an archive on this page that has File.hh, SendCompressedBuffer.hh and ReceiveCompressedBuffer.hh. direct.cc is an example of how to use the classes.


Brian Wood
http://webEbenezer.net
(651) 251-9384

Share this post


Link to post
Share on other sites
hplus0603    11356
File transfer is a solved problem. If you transfer files from a server, there is pretty much zero reason to not use HTTP. If your business suddenly scales, HTTP lets you use off-the-shelf CDNs, where the bits are really fast and cheap to deliver compared to hosting your own. And before you get to that point, you can use S3 to deliver the bits if you don't want to do it yourself.

Most languages have great libraries for making HTTP requests, too. It's really easy in Python/javascript/.NET/whatever.

Share this post


Link to post
Share on other sites
ValMan    466
File transfer from server is really easy to do, if you are already running an HTTP server. I wrote a simple file download routine just last week:

1. Include wininet.h and link to wininet.lib
2. Call InternetOpen to initialize WinInet library, save returned handle
3. Call InternetOpenUrl passing url and any necessary flags (see docs)
4. Get size of file if server supports that using HttpQueryInfo. If unavailable, cal still download blindly (function below doesn't)
5. Allocate buffer and loop, calling InternetReadFile until done. I chose to allocate buffer of file size rather than chunk size
6. Clean up, calling InternetCloseHandle on the file, then on WinInet library.
7. If you want to display a progress bar and a percentage label, insert additional code inside the reading loop to do that, as well as to process messages. Otherwise your code will not be async.


BOOL DownloadSync(LPCSTR pszUrl, LPBYTE* ppOutDest, LPDWORD pdwOutDestSize, BOOL bNullTerminate)
{
/* Initialize WinInet library */

HINTERNET hOpen = InternetOpen(SZ_APPTITLE, 0, NULL, 0, 0);

if(NULL == hOpen)
return FALSE;

/* Open handle to input file */

HINTERNET hFile = InternetOpenUrl(hOpen, pszUrl, NULL, 0, INTERNET_FLAG_RELOAD, 0);

if(NULL == hFile)
{
InternetCloseHandle(hOpen);
return FALSE;
}

/* Get size of input file - required */

DWORD dwBufSize = sizeof(DWORD);
DWORD dwFileSize = 0;

HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, (LPVOID)&dwFileSize, &dwBufSize, 0);

if(GetLastError() == ERROR_HTTP_HEADER_NOT_FOUND)
{
InternetCloseHandle(hFile);
InternetCloseHandle(hOpen);

return FALSE;
}

/* Allocate buffer */

LPBYTE pbBuffer = malloc(TRUE == bNullTerminate ? dwFileSize + 1 : dwFileSize);

if(NULL == pbBuffer)
{
InternetCloseHandle(hFile);
InternetCloseHandle(hOpen);

return FALSE;
}

ZeroMemory(pbBuffer, TRUE == bNullTerminate ? dwFileSize + 1 : dwFileSize);

/* Read file from connection until complete */

const DWORD DW_CHUNK_SIZE = 1000;
DWORD dwBytesRemaining = dwFileSize;
DWORD dwBytesToRead = 0;
DWORD dwBytesRead = 0;
LPBYTE pbBufferOffset = pbBuffer;

do
{
dwBytesToRead = min(DW_CHUNK_SIZE, dwBytesRemaining);

InternetReadFile(hFile, (LPVOID)pbBufferOffset, dwBytesToRead, &dwBytesRead);

dwBytesRemaining -= dwBytesRead;

pbBufferOffset += dwBytesRead;

} while (dwBytesRemaining > 0);

InternetCloseHandle(hFile);
InternetCloseHandle(hOpen);

*ppOutDest = pbBuffer;
*pdwOutDestSize = dwFileSize + 1;

return TRUE;
}


BOOL DownloadSyncFile(LPCSTR pszUrl, LPCSTR pszDestPath)
{
HANDLE hDest = CreateFile(pszDestPath, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, 0, NULL);

if(INVALID_HANDLE_VALUE == hDest)
return FALSE;

LPBYTE pbBuffer = NULL;
DWORD dwSize = 0;

if(DownloadSync(pszUrl, &pbBuffer, &dwSize, FALSE) != TRUE)
return FALSE;

DWORD dwWritten = 0;

WriteFile(hDest, (LPCVOID)pbBuffer, dwSize, &dwWritten, NULL);

if(dwWritten != dwSize)
return FALSE;

CloseHandle(hDest);

return TRUE;
}

Share this post


Link to post
Share on other sites
wood_brian    193
Quote:
Original post by hplus0603
File transfer is a solved problem. If you transfer files from a server, there is pretty much zero reason to not use HTTP. If your business suddenly scales, HTTP lets you use off-the-shelf CDNs, where the bits are really fast and cheap to deliver compared to hosting your own. And before you get to that point, you can use S3 to deliver the bits if you don't want to do it yourself.

Most languages have great libraries for making HTTP requests, too. It's really easy in Python/javascript/.NET/whatever.


I've only checked into CDNs for about 15 minutes, but don't see how that is relevant to what I'm working on. Are CDNs used for things like movies and music? What is S3? I looked at libcurl months ago, but decided against using it. In some areas -- compression -- I've compromised and used C libraries, but in general I want to use mostly C++ and that sometimes means reinventing the wheel.

I'm going to post a question about converting the results of the Windows GetFileTime function to a time_t on the general programming board in a few minutes. I mention it
in case someone here knows about that.

Brian Wood
http://webEbenezer.net
(651) 251-9384

Share this post


Link to post
Share on other sites
demonkoryu    980
Quote:
Original post by wood_brian
I've only checked into CDNs for about 15 minutes, but don't see how that is relevant to what I'm working on. Are CDNs used for things like movies and music?


He talked about scaling. With CDN's, you upload files to them, where they are then distributed around their network. The effect is, people download automatically from their nearest edge servers, thus traffic is distributed and transfers come from the nearest high-speed location.
They are basically distributed HTTP file servers.

Share this post


Link to post
Share on other sites
wood_brian    193
Quote:
Original post by Konfusius

He talked about scaling. With CDN's, you upload files to them, where they are then distributed around their network. The effect is, people download automatically from their nearest edge servers, thus traffic is distributed and transfers come from the nearest high-speed location.
They are basically distributed HTTP file servers.


I understand that, but still am not sure if it is applicable to what I'm working on. I have input and output files in my stuff. It doesn't make sense to me to use a CDN for either of those. Using a CDN to host files that many people are interested in makes sense to me.

Brian Wood
http://webEbenezer.net
(651) 251-9384

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this