Python auto patcher

Started by
9 comments, last by Developer_X 14 years, 8 months ago
Hi, I am working on a patcher for my game and decided to give Python a try. Currently it's working well, but I'd like some feedback on my general plan. 1. Python script gets the last modified date for every local client file, storing it in a file. The end file will look something like this: 'file1.f-20090811' 'file2.f-20090809' etc. 2. Send it to the web server (via a POST, I think?) 3. Web server receives file. 4. Web server parses all appropriate files on the server to see if the web server's version is more up to date (the server has every file up to date - when I make changes to a file, I upload it to the web server). 5. The web server compresses the files into some format, making it one large(r) download, and allows the client to download the file. 6. The python script will then uncompress and place the files in the right spot. Basically I am having issue with number 2 and number 3, mostly because I have never tried anything like this. Do I send an HTTP post message? Am I supposed to be using anything fancy? Will PHP be a viable candidate to receive the file, parse it, and group the files into one download? Any feedback is appreciated, thanks!
Advertisement
Generally, a lot of patchers work the other way around:

1. Client connects to web server to request a list of files (version, size, CRCs, whatever is most useful).

2. Client locally checks its files against the list (keeps workload on client side).

3. Client requests the files that "need updating" from the server one at a time.

By keeping it client driven, you can simplify your web server logic to simply host the files and the client will download what it needs. I've not seen many games where the server is responsible for the patch work like you described. It's doable, but adds a lot of complexity as you already see.

So on the server, the setup might be:
Root+ version.txt [contains 0.0.3|0.0.2|0.0.1]- 0.0.1-- filelist.txt [contains directory contents]-- Readme.txt.zip- 0.0.2-- filelist.txt [contains directory contents]-- logo.png.zip-- Readme.txt.zip- 0.0.3-- filelist.txt [contains directory contents]-- client.exe.zip
So the client would start by downloading the version.txt and check the version to its own. It would then sequentially access the next version's filelist.txt to get the list of files it needs to download and replace. When the version finally reaches the latest, you know the patch is complete.

There are variations of how you want to set this up, but I just gave one example I've seen. The versions to update are stored in the version.txt simply so the clients knows what to access next. As you phase out updates, the client would need logic to tell the user they need to redownload the latest package for a new "base client".
The problem actually arose because I a built a patcher doing exactly that (well mostly) but would routinely get disconnected by the web server.

If I do it that way though, why would I ever need to keep track of the right version? Wouldn't I just alter the 'version.txt' with more up to date times, and the client would request a download? (EDIT HERE) Basically, is it feasible to get a truly auto-patcher, where I dont do anything but upload the files (and maybe update the times file)?

Since you brought this up, is there a way around getting forcefully disconnected from the web host when requesting lots of downloads? Basically after it downloads X files, it starts to slow down and then stop, and after some time it spits out an error (I cant seem to replicate it now, but its something that conveys the server has stopped the connection).

Edit 2: Here's the actual error

URLError: <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>

[Edited by - Crazyfool on August 15, 2009 11:32:18 AM]
What does your download code look like? The way I'd envision this working is that the client would make a GET request to the server, which would respond with a list of files, their download URLs, and their CRCs. The client would then compare the file CRCs to the local versions, and request downloads for the ones that don't match. Each of these download requests, as well as the initial GET request, would be a separate server request.

If you're somehow keeping the server connection open while you do this work, then you're liable to run into problems as webservers have a maximum amount of time they keep connections open without data being transmitted (IIRC the default for Apache is 1 minute), after which they're assumed faulty in some way and are disconnected.
Jetblade: an open-source 2D platforming game in the style of Metroid and Castlevania, with procedurally-generated levels
Well, the downloading starts pretty quickly and continues till the connecting ending.

The code looks something like this for downloading files:
def checkDownload(filename):    url = 'site_address' + filename    webFile = urllib2.urlopen(url)    if os.path.exists(filename):        localDate = time.strftime("%Y%m%d", time.gmtime(os.stat(filename).st_mtime))    else:        localDate = 0    webDate = time.strftime("%Y%m%d", time.gmtime(time.mktime(webFile.info().getdate('Last-Modified'))))    if webDate > localDate:        print 'Downloading', filename        localFile = open(filename, 'wb')        localFile.write(webFile.read())        localFile.close()    webFile.close()
Quote:Original post by Crazyfool
If I do it that way though, why would I ever need to keep track of the right version? Wouldn't I just alter the 'version.txt' with more up to date times, and the client would request a download?


There are many approaches to handling updates, I've seen all sorts of different approaches taken but that was just one specific variation. Some people just maintain a "current" file set and you simply download those to stay up to date. Others take a more organized approach and do the version numbers. The benefits of version control are more on the side of debugging as you can go back and track down bugs or figure out why something changed along the way.

Quote:Basically, is it feasible to get a truly auto-patcher, where I dont do anything but upload the files (and maybe update the times file)?


Sure, it just depends how complex you want your system. You could write a simple version manager system in PHP or ASP that would allow you to upload files and automatically generate everything you need, but that's on the complex side of things. You could just manage the file yourself which is more work but keeps things simple.

Quote:Since you brought this up, is there a way around getting forcefully disconnected from the web host when requesting lots of downloads? Basically after it downloads X files, it starts to slow down and then stop, and after some time it spits out an error (I cant seem to replicate it now, but its something that conveys the server has stopped the connection).


That sounds like a web server specific issue in response to how your program works. I'm not really knowledge in that area of setting up web servers or configuring, but the API that you are using to download files might be doing something the server doesn't like. If you are using a regular web server, you should be able to download large files from it without any problems. If your host is disconnecting the clients or it gets messed up along the way, I'd assume there is an issue with your code.

I'm not knowledgeable in python either, so perhaps you could post your python code and someone who knows network programming as well as python can have a look at it? Also are you setting up your own web server or using generic hosting from a 3rd party. That kind of information is helpful for trying to track down that issue.

I'm not sure how much this might help you, but here is a complete simple C++/Win32 example I've whipped up of one of the easiest ways to go about this task. It is a live example and will download two files from my GameDev space.

#include <windows.h>#include <urlmon.h>#include <fstream>#include <sstream>#include <string>#include <vector>#pragma comment(lib, "urlmon.lib")// Static CRC tableDWORD s_arrdwCrc32Table[256] ={	0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA,	0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3,	0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988,	0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91,	0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE,	0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7,	0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC,	0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5,	0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172,	0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B,	0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940,	0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59,	0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116,	0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F,	0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924,	0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D,	0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A,	0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433,	0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818,	0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01,	0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E,	0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457,	0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C,	0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65,	0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2,	0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB,	0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0,	0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9,	0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086,	0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F,	0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4,	0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD,	0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A,	0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683,	0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8,	0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1,	0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE,	0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7,	0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC,	0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5,	0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252,	0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B,	0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60,	0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79,	0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236,	0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F,	0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04,	0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D,	0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A,	0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713,	0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38,	0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21,	0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E,	0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777,	0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C,	0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45,	0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2,	0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB,	0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0,	0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9,	0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6,	0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF,	0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94,	0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D,};// Calculate the crc of a byte sequenceinline void CalcCrc32(const BYTE byte, DWORD &dwCrc32){	dwCrc32 = ((dwCrc32) >> 8) ^ s_arrdwCrc32Table[(byte) ^ ((dwCrc32) & 0x000000FF)];}// Calculate the CRC of a filebool GetFileCrc(LPCTSTR szFilename, DWORD & dwCrc32){	//DWORD dwErrorCode = NO_ERROR;	HANDLE hFile = NULL;	dwCrc32 = 0xFFFFFFFF;	hFile = CreateFile(szFilename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_ARCHIVE | FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_READONLY | FILE_ATTRIBUTE_SYSTEM | FILE_FLAG_SEQUENTIAL_SCAN, NULL);	if(hFile == INVALID_HANDLE_VALUE)	{		return false;	}	else	{		BYTE buffer[4096] = {0};		DWORD dwBytesRead = 0;		DWORD dwLoop = 0;		BOOL bSuccess = ReadFile(hFile, buffer, sizeof(buffer), &dwBytesRead, NULL);		while(bSuccess && dwBytesRead)		{			for(dwLoop = 0; dwLoop < dwBytesRead; dwLoop++)			{				CalcCrc32(buffer[dwLoop], dwCrc32);			}			bSuccess = ReadFile(hFile, buffer, sizeof(buffer), &dwBytesRead, NULL);		}	}	CloseHandle(hFile);	dwCrc32 = ~dwCrc32;	return true;}// Returns an integer from a hex stringint HexStringToInteger(const char * str){	size_t length = strlen(str);	char * strUpper = new char[length + 1];	memcpy(strUpper, str, length);	strUpper[length] = 0;	for(size_t x = 0; x < length; ++x)		strUpper[x] = (char)toupper(strUpper[x]);	// http://www.codeproject.com/string/hexstrtoint.asp	struct CHexMap	{		CHAR chr;		int value;	};	const int HexMapL = 16;	CHexMap HexMap[HexMapL] =	{		{'0', 0}, {'1', 1},		{'2', 2}, {'3', 3},		{'4', 4}, {'5', 5},		{'6', 6}, {'7', 7},		{'8', 8}, {'9', 9},		{'A', 10}, {'B', 11},		{'C', 12}, {'D', 13},		{'E', 14}, {'F', 15}	};	CHAR *mstr = strUpper;	CHAR *s = mstr;	int result = 0;	if (*s == '0' && *(s + 1) == 'X') s += 2;	bool firsttime = true;	while (*s != '\0')	{		bool found = false;		for (int i = 0; i < HexMapL; i++)		{			if (*s == HexMap.chr)			{				if (!firsttime) 					result <<= 4;				result |= HexMap.value;				found = true;				break;			}		}		if (!found) break;		s++;		firsttime = false;	}	delete [] strUpper;	return result;}// Tokenizes a string into a vectorstd::vector<std::string> TokenizeString(const std::string& str, const std::string& delim){	// http://www.gamedev.net/community/forums/topic.asp?topic_id=381544#TokenizeString	using namespace std;	vector<string> tokens;	size_t p0 = 0, p1 = string::npos;	while(p0 != string::npos)	{		p1 = str.find_first_of(delim, p0);		if(p1 != p0)		{			string token = str.substr(p0, p1 - p0);			tokens.push_back(token);		}		p0 = str.find_first_not_of(delim, p1);	}	return tokens;}int main(int argc, char * argv[]){	char baseDir[256] = {0};	GetCurrentDirectory(256, baseDir);	// Remove the patch file if it alreacy exists	std::string patchFileName = baseDir;	patchFileName += "\\patch.txt";	DeleteFileA(patchFileName.c_str());	// Try to download the patch file list	HRESULT result = URLDownloadToFileA(NULL, "http://members.gamedev.net/edxLabs/autopatch/patch.txt", patchFileName.c_str(), 0, NULL);	if(result != S_OK)	{		MessageBoxA(0, "Could not obtain the patch list.", "Fatal Error", MB_ICONERROR);		return 0;	}	std::ifstream inFile(patchFileName.c_str());	if(inFile.is_open() == false)	{		MessageBoxA(0, "Could not open the the patch.txt file.", "Fatal Error", MB_ICONERROR);		return 0;	}	// Create a tmp dir for files to download into	std::string tmpDirectory = baseDir;	tmpDirectory += "\\tmp";	CreateDirectory(tmpDirectory.c_str(), 0);	std::string buffer;	while(std::getline(inFile, buffer))	{		// Parse the input file		std::vector<std::string> parts = TokenizeString(buffer, "\t");		if(parts.size() != 3)		{			printf("The entry \"%s\" is invalid.\n", buffer.c_str());			continue;		}		std::string filename = parts[0];		std::string filecrc = parts[1];		std::string filepath = parts[2];		// Build a path toe the real file		std::string fpath = baseDir;		fpath += "\\";		fpath += filepath;		fpath += "\\";		fpath += filename;		bool doDownload = true;		// If the file exists, try to get the CRC		DWORD crc;		if(GetFileCrc(fpath.c_str(), crc) == true)		{			DWORD expectedCRC = HexStringToInteger(filecrc.c_str());			// If the CRCs match, we do not need to update			if(expectedCRC == crc)			{				doDownload = false;				printf("File %s does not need to be updated.\n", filename.c_str());			}		}		// If we need to download this file		if(doDownload)		{			// Build the path to the file to download			std::string baseUrl = "http://members.gamedev.net/edxLabs/autopatch/";			baseUrl += filename;			// Build a path to the tmp file we are about to download			std::string tmpFilename = tmpDirectory;			tmpFilename += "\\";			tmpFilename += filename;			// Try to download the file, this function blocks! You need to use the API more fully			// for better uses.			HRESULT result = URLDownloadToFileA(NULL, baseUrl.c_str(), tmpFilename.c_str(), 0, NULL);			if(result != S_OK)			{				printf("Could not download the updated %s file.\n", filename.c_str());				continue;			}			// We need to make sure the final file path exists, so we just recreate them			// for each file. This is not efficient for a lot of files, but this is			// just a simple example...			std::vector<std::string> paths = TokenizeString(filepath, "\\/");			for(size_t x = 0; x < paths.size(); ++x)			{				std::stringstream builder;				builder << baseDir << "\\";				for(size_t y = 0; y < x; ++y)				{					builder << paths[y] << "\\";				}				builder << paths[x];				CreateDirectory(builder.str().c_str(), 0);			}			// Remove the old file (should rename just in case the MoveFile fails!)			DeleteFileA(fpath.c_str());			// Try and copy over the new file			if(MoveFile(tmpFilename.c_str(), fpath.c_str()) == FALSE)			{				printf("Could not update: %s\n", filename.c_str());				// Remove the temp file				DeleteFile(tmpFilename.c_str());				// Try the next file				continue;			}			else			{				printf("Updating %s\n", filename.c_str());			}		}	}	inFile.close();	DeleteFileA(patchFileName.c_str());	RemoveDirectory(tmpDirectory.c_str());	return 0;}


This example makes no use of version numbers and is more along the lines of what you were asking about. If you want to try it yourself on your own server, all you need to do is create a "patch.txt" file that has a format of: filename<tab>crc<tab>destination path and then have the files in the same directory.

That's about as simple as you can get it (using Win32 at least). Just to reiterate the process is:
1. Client obtains a list of files
2. Client checks each file against its own
3. For each file in the list:
3a If the file is missing, download it to a tmp folder
3b. If the file exists and CRC matches, skip it
3c. If the file exists and CRC does not match, download it to a tmp folder
3d. Remove/Rename the original file and then replace it with the new file
4. Process is completed. If there were errors exit and tell the user to try again.

I'm suspecting your specific problems are on the Python side rather than the logic side. As mentioned before, can you give more information on the specific Python code or APIs you are using for the task as well as the web server information?

[Edit] Didn't see your post about the python code, but I'd not be much help suggesting alternatives in Python.
I really appreciate the help. I actually was initially going to go the C++ route before, but I had issues with getting the particular download method to work. I will definitely look at your code and adapt to my needs.

Thanks again!
Alright, this code has been EXTRAORDINARILY helpful, and I TRULY appreciate it.

My only concern is letter case. Is there any way to make the downloads case-insensitive without attempting many different versions of the file? (For instance, a lot of my files are .png, but some are .PNG, and if a .png file fails, I can simply retry as .PNG).

Thanks again!
Quote:Original post by Crazyfool
My only concern is letter case. Is there any way to make the downloads case-insensitive without attempting many different versions of the file? (For instance, a lot of my files are .png, but some are .PNG, and if a .png file fails, I can simply retry as .PNG).


On Windows, files are not case sensitive but on *nix they are. You might be able to work up a convoluted solution using apache and mod_rewrite or using your htaccess, but generally, if you are using *nix, you just have to make sure you set the names yourself correctly.

I'd just go to manually renaming all the files yourself to lower case or better yet, create a Windows program to do it and save yourself the headaches! For example, this is where a higher level language is very handy since you can just do something like this, a small VB.net program that you can compile and run in your file directory to easily rename the files' extensions to lower case.
Module Module1    Sub Main()        For Each foundFile As String In My.Computer.FileSystem.GetFiles(My.Computer.FileSystem.CurrentDirectory)            Dim strFile            Dim lastIndex            Dim tmpFile            Dim finalFile            Dim tmpParts            ' Breakup the file by path separators to get the file title            strFile = Split(foundFile, "\")            lastIndex = UBound(strFile)            finalFile = strFile(lastIndex).ToLower()            ' Get the extension for the title and make it lower case            tmpParts = Split(finalFile, ".")            lastIndex = UBound(tmpParts)            tmpParts(lastIndex) = tmpParts(lastIndex).ToLower()            ' Rebuild the string by adding back the .'s            finalFile = String.Join(".", tmpParts)            ' Store the temp name of the file since Windows needs             ' unique file names to rename properly            tmpFile = finalFile + "_"            Try                ' Rename the current file into the tmp file                My.Computer.FileSystem.RenameFile(foundFile, tmpFile)                ' Rename the tmp file into the final file                My.Computer.FileSystem.RenameFile(tmpFile, finalFile)                ' Status                System.Console.WriteLine("Successfully renamed the file " + finalFile)            Catch ex As Exception                ' Status                System.Console.WriteLine("Could not rename the file " + foundFile)            End Try        Next    End SubEnd Module


You could also take it one step further and just make it generate the patch.txt file for you automatically. In that case, you'd not even need to rename the files. You just would subtract out the base path from the current path to get the relative path, find a CRC32 function, and simply write out the final file.

I'd suggest an approach like that because you write the tool once, run it in your root patch directory, and just have to upload the file to your webserver. Heck, you can even just add the FTP code to the program itself and make it 100% automated. All you have to do is run the exe [smile]

So, there's lots of cool things to do to make life easier, I'd defeintly suggest the route of making your own small utility in a language of your choice for it. Whatever makes the task as easy as possible. I'm not even a VB.net person but you can see how simple .net makes things such as this, so choosing the right tools for the job makes all this stuff more fun!

Just some ideas, good luck!
Thanks a bundle. I think this has been the 100th time you've help me.

Edit: And I agree about the higher level language. While my solution is going to be in C++ (thanks to you), playing around in Python was quite fun and enjoyable. It also taught me some things, and I think I could have easily done it with Python had I stuck with it (Compare my 1-2 days of Python experience and being able to get something to at least marginally work to the years of C++ experience and having difficulties ;) )

This topic is closed to new replies.

Advertisement