Small file transfers?
Author
Anyone know of a very small and/or lightweight API/lib for doing file transfers?
Basically, need to do a few small file transfers (100k ish) and I'm uncertain of a decent starting point.
I want to send map information to a client from the server... packets might work, but it would take a bunch... O.o
What protocol? TCP? UDP?
You'll probably be quickest using whatever you're using now for a relatively small file - If you're using UDP, some sort of basic reliable packet system would work quite well.
You'll probably be quickest using whatever you're using now for a relatively small file - If you're using UDP, some sort of basic reliable packet system would work quite well.
Author
TCP.
I'm using a pretty basic, but robust system, which is a net event system (XML marshaling).
I mean, 100k isn't much at the end of the day. I'm having trouble deciding if I want to send global maps everytime a player logs in, or if I just want to send the map data as they switch maps.
It would be easier just to drop a map file into the map directory for the client. It might be more footwork to send a couple map packets with map data in it, at least, it would be kind of wasteful, data-transmission wise.
I don't know. I figured there's got to be some small module out there that just does basic TCP file-xfers?
Cross platform preferred, but the primary system would be win32.
I'm using a pretty basic, but robust system, which is a net event system (XML marshaling).
I mean, 100k isn't much at the end of the day. I'm having trouble deciding if I want to send global maps everytime a player logs in, or if I just want to send the map data as they switch maps.
It would be easier just to drop a map file into the map directory for the client. It might be more footwork to send a couple map packets with map data in it, at least, it would be kind of wasteful, data-transmission wise.
I don't know. I figured there's got to be some small module out there that just does basic TCP file-xfers?
Cross platform preferred, but the primary system would be win32.
I was going to suggest this jokingly but if it works for you why not.
You could bundle wget (windows build) with your client, stick the map files on a webserver, and run a system command to pull down those files to the map folder.
Sure it requires calling an external program, but wget is fairly robust and this will require minimal work from you.
[edit] You could even use ftp thats supplied with most windows versions(short of 3.11 and below) to pull the files down from anonymous ftp ftp scripts[/edit]
You could bundle wget (windows build) with your client, stick the map files on a webserver, and run a system command to pull down those files to the map folder.
Sure it requires calling an external program, but wget is fairly robust and this will require minimal work from you.
[edit] You could even use ftp thats supplied with most windows versions(short of 3.11 and below) to pull the files down from anonymous ftp ftp scripts[/edit]
If the file transfer is from some server you know, then use HTTP, and use whatever HTTP client is convenient for your language. In C/C++, there's for example libcurl, or the ultra-small HTTP-GET.
If the file transfer needs to be fancy, like doing peer-to-peer or from untrusted sources with digital signatures, then you'd have to give us a little more information for us to answer the question. It's very unlikely that rolling your own, custom protocol on top of TCP is really the right answer, except in certain narrow cases.
If the file transfer needs to be fancy, like doing peer-to-peer or from untrusted sources with digital signatures, then you'd have to give us a little more information for us to answer the question. It's very unlikely that rolling your own, custom protocol on top of TCP is really the right answer, except in certain narrow cases.
Author
Yeah, it would be a trusted source (server). And nothing fancy required.
Basically it's just downloading the most current map versions so the client has the most up to date map the server has.
Then again the map data is quite simple, at this point I'm wondering if just streaming it as packets might be feasible. Then again, if texture/tile data doesn't change regularly, why waste the bandwidth... We're talking maybe 20 tiles (3 ints for data) in the buffer area...
Basically it's just downloading the most current map versions so the client has the most up to date map the server has.
Then again the map data is quite simple, at this point I'm wondering if just streaming it as packets might be feasible. Then again, if texture/tile data doesn't change regularly, why waste the bandwidth... We're talking maybe 20 tiles (3 ints for data) in the buffer area...
Quote:Original post by catch
TCP.
I'm using a pretty basic, but robust system, which is a net event system (XML marshaling).
I have an API that transfers files. I'm not sure how it would work with XML stuff. I've never tried it with that. There's an archive on this page that has File.hh, SendCompressedBuffer.hh and ReceiveCompressedBuffer.hh. direct.cc is an example of how to use the classes.
Brian Wood
http://webEbenezer.net
(651) 251-9384
File transfer is a solved problem. If you transfer files from a server, there is pretty much zero reason to not use HTTP. If your business suddenly scales, HTTP lets you use off-the-shelf CDNs, where the bits are really fast and cheap to deliver compared to hosting your own. And before you get to that point, you can use S3 to deliver the bits if you don't want to do it yourself.
Most languages have great libraries for making HTTP requests, too. It's really easy in Python/javascript/.NET/whatever.<br>
Most languages have great libraries for making HTTP requests, too. It's really easy in Python/javascript/.NET/whatever.<br>
File transfer from server is really easy to do, if you are already running an HTTP server. I wrote a simple file download routine just last week:
1. Include wininet.h and link to wininet.lib
2. Call InternetOpen to initialize WinInet library, save returned handle
3. Call InternetOpenUrl passing url and any necessary flags (see docs)
4. Get size of file if server supports that using HttpQueryInfo. If unavailable, cal still download blindly (function below doesn't)
5. Allocate buffer and loop, calling InternetReadFile until done. I chose to allocate buffer of file size rather than chunk size
6. Clean up, calling InternetCloseHandle on the file, then on WinInet library.
7. If you want to display a progress bar and a percentage label, insert additional code inside the reading loop to do that, as well as to process messages. Otherwise your code will not be async.
1. Include wininet.h and link to wininet.lib
2. Call InternetOpen to initialize WinInet library, save returned handle
3. Call InternetOpenUrl passing url and any necessary flags (see docs)
4. Get size of file if server supports that using HttpQueryInfo. If unavailable, cal still download blindly (function below doesn't)
5. Allocate buffer and loop, calling InternetReadFile until done. I chose to allocate buffer of file size rather than chunk size
6. Clean up, calling InternetCloseHandle on the file, then on WinInet library.
7. If you want to display a progress bar and a percentage label, insert additional code inside the reading loop to do that, as well as to process messages. Otherwise your code will not be async.
BOOL DownloadSync(LPCSTR pszUrl, LPBYTE* ppOutDest, LPDWORD pdwOutDestSize, BOOL bNullTerminate){ /* Initialize WinInet library */ HINTERNET hOpen = InternetOpen(SZ_APPTITLE, 0, NULL, 0, 0); if(NULL == hOpen) return FALSE; /* Open handle to input file */ HINTERNET hFile = InternetOpenUrl(hOpen, pszUrl, NULL, 0, INTERNET_FLAG_RELOAD, 0); if(NULL == hFile) { InternetCloseHandle(hOpen); return FALSE; } /* Get size of input file - required */ DWORD dwBufSize = sizeof(DWORD); DWORD dwFileSize = 0; HttpQueryInfo(hFile, HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER, (LPVOID)&dwFileSize, &dwBufSize, 0); if(GetLastError() == ERROR_HTTP_HEADER_NOT_FOUND) { InternetCloseHandle(hFile); InternetCloseHandle(hOpen); return FALSE; } /* Allocate buffer */ LPBYTE pbBuffer = malloc(TRUE == bNullTerminate ? dwFileSize + 1 : dwFileSize); if(NULL == pbBuffer) { InternetCloseHandle(hFile); InternetCloseHandle(hOpen); return FALSE; } ZeroMemory(pbBuffer, TRUE == bNullTerminate ? dwFileSize + 1 : dwFileSize); /* Read file from connection until complete */ const DWORD DW_CHUNK_SIZE = 1000; DWORD dwBytesRemaining = dwFileSize; DWORD dwBytesToRead = 0; DWORD dwBytesRead = 0; LPBYTE pbBufferOffset = pbBuffer; do { dwBytesToRead = min(DW_CHUNK_SIZE, dwBytesRemaining); InternetReadFile(hFile, (LPVOID)pbBufferOffset, dwBytesToRead, &dwBytesRead); dwBytesRemaining -= dwBytesRead; pbBufferOffset += dwBytesRead; } while (dwBytesRemaining > 0); InternetCloseHandle(hFile); InternetCloseHandle(hOpen); *ppOutDest = pbBuffer; *pdwOutDestSize = dwFileSize + 1; return TRUE;}BOOL DownloadSyncFile(LPCSTR pszUrl, LPCSTR pszDestPath){ HANDLE hDest = CreateFile(pszDestPath, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, 0, NULL); if(INVALID_HANDLE_VALUE == hDest) return FALSE; LPBYTE pbBuffer = NULL; DWORD dwSize = 0; if(DownloadSync(pszUrl, &pbBuffer, &dwSize, FALSE) != TRUE) return FALSE; DWORD dwWritten = 0; WriteFile(hDest, (LPCVOID)pbBuffer, dwSize, &dwWritten, NULL); if(dwWritten != dwSize) return FALSE; CloseHandle(hDest); return TRUE;}Quote:Original post by hplus0603
File transfer is a solved problem. If you transfer files from a server, there is pretty much zero reason to not use HTTP. If your business suddenly scales, HTTP lets you use off-the-shelf CDNs, where the bits are really fast and cheap to deliver compared to hosting your own. And before you get to that point, you can use S3 to deliver the bits if you don't want to do it yourself.
Most languages have great libraries for making HTTP requests, too. It's really easy in Python/javascript/.NET/whatever.
I've only checked into CDNs for about 15 minutes, but don't see how that is relevant to what I'm working on. Are CDNs used for things like movies and music? What is S3? I looked at libcurl months ago, but decided against using it. In some areas -- compression -- I've compromised and used C libraries, but in general I want to use mostly C++ and that sometimes means reinventing the wheel.
I'm going to post a question about converting the results of the Windows GetFileTime function to a time_t on the general programming board in a few minutes. I mention it
in case someone here knows about that.
Brian Wood
http://webEbenezer.net
(651) 251-9384
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
