Public Group

# Python auto patcher

This topic is 3257 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

##### Share on other sites
Generally, a lot of patchers work the other way around:

1. Client connects to web server to request a list of files (version, size, CRCs, whatever is most useful).

2. Client locally checks its files against the list (keeps workload on client side).

3. Client requests the files that "need updating" from the server one at a time.

By keeping it client driven, you can simplify your web server logic to simply host the files and the client will download what it needs. I've not seen many games where the server is responsible for the patch work like you described. It's doable, but adds a lot of complexity as you already see.

So on the server, the setup might be:
Root+ version.txt [contains 0.0.3|0.0.2|0.0.1]- 0.0.1-- filelist.txt [contains directory contents]-- Readme.txt.zip- 0.0.2-- filelist.txt [contains directory contents]-- logo.png.zip-- Readme.txt.zip- 0.0.3-- filelist.txt [contains directory contents]-- client.exe.zip
So the client would start by downloading the version.txt and check the version to its own. It would then sequentially access the next version's filelist.txt to get the list of files it needs to download and replace. When the version finally reaches the latest, you know the patch is complete.

There are variations of how you want to set this up, but I just gave one example I've seen. The versions to update are stored in the version.txt simply so the clients knows what to access next. As you phase out updates, the client would need logic to tell the user they need to redownload the latest package for a new "base client".

##### Share on other sites
The problem actually arose because I a built a patcher doing exactly that (well mostly) but would routinely get disconnected by the web server.

If I do it that way though, why would I ever need to keep track of the right version? Wouldn't I just alter the 'version.txt' with more up to date times, and the client would request a download? (EDIT HERE) Basically, is it feasible to get a truly auto-patcher, where I dont do anything but upload the files (and maybe update the times file)?

Since you brought this up, is there a way around getting forcefully disconnected from the web host when requesting lots of downloads? Basically after it downloads X files, it starts to slow down and then stop, and after some time it spits out an error (I cant seem to replicate it now, but its something that conveys the server has stopped the connection).

Edit 2: Here's the actual error

URLError: <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>

[Edited by - Crazyfool on August 15, 2009 11:32:18 AM]

##### Share on other sites
What does your download code look like? The way I'd envision this working is that the client would make a GET request to the server, which would respond with a list of files, their download URLs, and their CRCs. The client would then compare the file CRCs to the local versions, and request downloads for the ones that don't match. Each of these download requests, as well as the initial GET request, would be a separate server request.

If you're somehow keeping the server connection open while you do this work, then you're liable to run into problems as webservers have a maximum amount of time they keep connections open without data being transmitted (IIRC the default for Apache is 1 minute), after which they're assumed faulty in some way and are disconnected.

##### Share on other sites

def checkDownload(filename):    url = 'site_address' + filename    webFile = urllib2.urlopen(url)    if os.path.exists(filename):        localDate = time.strftime("%Y%m%d", time.gmtime(os.stat(filename).st_mtime))    else:        localDate = 0    webDate = time.strftime("%Y%m%d", time.gmtime(time.mktime(webFile.info().getdate('Last-Modified'))))    if webDate > localDate:        print 'Downloading', filename        localFile = open(filename, 'wb')        localFile.write(webFile.read())        localFile.close()    webFile.close()

##### Share on other sites
Quote:
 Original post by CrazyfoolIf I do it that way though, why would I ever need to keep track of the right version? Wouldn't I just alter the 'version.txt' with more up to date times, and the client would request a download?

There are many approaches to handling updates, I've seen all sorts of different approaches taken but that was just one specific variation. Some people just maintain a "current" file set and you simply download those to stay up to date. Others take a more organized approach and do the version numbers. The benefits of version control are more on the side of debugging as you can go back and track down bugs or figure out why something changed along the way.

Quote:
 Basically, is it feasible to get a truly auto-patcher, where I dont do anything but upload the files (and maybe update the times file)?

Sure, it just depends how complex you want your system. You could write a simple version manager system in PHP or ASP that would allow you to upload files and automatically generate everything you need, but that's on the complex side of things. You could just manage the file yourself which is more work but keeps things simple.

Quote:
 Since you brought this up, is there a way around getting forcefully disconnected from the web host when requesting lots of downloads? Basically after it downloads X files, it starts to slow down and then stop, and after some time it spits out an error (I cant seem to replicate it now, but its something that conveys the server has stopped the connection).

That sounds like a web server specific issue in response to how your program works. I'm not really knowledge in that area of setting up web servers or configuring, but the API that you are using to download files might be doing something the server doesn't like. If you are using a regular web server, you should be able to download large files from it without any problems. If your host is disconnecting the clients or it gets messed up along the way, I'd assume there is an issue with your code.

I'm not knowledgeable in python either, so perhaps you could post your python code and someone who knows network programming as well as python can have a look at it? Also are you setting up your own web server or using generic hosting from a 3rd party. That kind of information is helpful for trying to track down that issue.

I'm not sure how much this might help you, but here is a complete simple C++/Win32 example I've whipped up of one of the easiest ways to go about this task. It is a live example and will download two files from my GameDev space.

#include <windows.h>#include <urlmon.h>#include <fstream>#include <sstream>#include <string>#include <vector>#pragma comment(lib, "urlmon.lib")// Static CRC tableDWORD s_arrdwCrc32Table[256] ={	0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA,	0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3,	0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988,	0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91,	0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE,	0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7,	0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC,	0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5,	0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172,	0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B,	0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940,	0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59,	0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116,	0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F,	0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924,	0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D,	0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A,	0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433,	0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818,	0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01,	0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E,	0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457,	0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C,	0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65,	0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2,	0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB,	0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0,	0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9,	0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086,	0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F,	0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4,	0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD,	0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A,	0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683,	0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8,	0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1,	0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE,	0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7,	0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC,	0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5,	0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252,	0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B,	0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60,	0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79,	0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236,	0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F,	0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04,	0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D,	0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A,	0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713,	0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38,	0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21,	0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E,	0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777,	0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C,	0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45,	0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2,	0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB,	0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0,	0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9,	0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6,	0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF,	0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94,	0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D,};// Calculate the crc of a byte sequenceinline void CalcCrc32(const BYTE byte, DWORD &dwCrc32){	dwCrc32 = ((dwCrc32) >> 8) ^ s_arrdwCrc32Table[(byte) ^ ((dwCrc32) & 0x000000FF)];}// Calculate the CRC of a filebool GetFileCrc(LPCTSTR szFilename, DWORD & dwCrc32){	//DWORD dwErrorCode = NO_ERROR;	HANDLE hFile = NULL;	dwCrc32 = 0xFFFFFFFF;	hFile = CreateFile(szFilename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_ARCHIVE | FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_READONLY | FILE_ATTRIBUTE_SYSTEM | FILE_FLAG_SEQUENTIAL_SCAN, NULL);	if(hFile == INVALID_HANDLE_VALUE)	{		return false;	}	else	{		BYTE buffer[4096] = {0};		DWORD dwBytesRead = 0;		DWORD dwLoop = 0;		BOOL bSuccess = ReadFile(hFile, buffer, sizeof(buffer), &dwBytesRead, NULL);		while(bSuccess && dwBytesRead)		{			for(dwLoop = 0; dwLoop < dwBytesRead; dwLoop++)			{				CalcCrc32(buffer[dwLoop], dwCrc32);			}			bSuccess = ReadFile(hFile, buffer, sizeof(buffer), &dwBytesRead, NULL);		}	}	CloseHandle(hFile);	dwCrc32 = ~dwCrc32;	return true;}// Returns an integer from a hex stringint HexStringToInteger(const char * str){	size_t length = strlen(str);	char * strUpper = new char[length + 1];	memcpy(strUpper, str, length);	strUpper[length] = 0;	for(size_t x = 0; x < length; ++x)		strUpper[x] = (char)toupper(strUpper[x]);	// http://www.codeproject.com/string/hexstrtoint.asp	struct CHexMap	{		CHAR chr;		int value;	};	const int HexMapL = 16;	CHexMap HexMap[HexMapL] =	{		{'0', 0}, {'1', 1},		{'2', 2}, {'3', 3},		{'4', 4}, {'5', 5},		{'6', 6}, {'7', 7},		{'8', 8}, {'9', 9},		{'A', 10}, {'B', 11},		{'C', 12}, {'D', 13},		{'E', 14}, {'F', 15}	};	CHAR *mstr = strUpper;	CHAR *s = mstr;	int result = 0;	if (*s == '0' && *(s + 1) == 'X') s += 2;	bool firsttime = true;	while (*s != '\0')	{		bool found = false;		for (int i = 0; i < HexMapL; i++)		{			if (*s == HexMap.chr)			{				if (!firsttime) 					result <<= 4;				result |= HexMap.value;				found = true;				break;			}		}		if (!found) break;		s++;		firsttime = false;	}	delete [] strUpper;	return result;}// Tokenizes a string into a vectorstd::vector<std::string> TokenizeString(const std::string& str, const std::string& delim){	// http://www.gamedev.net/community/forums/topic.asp?topic_id=381544#TokenizeString	using namespace std;	vector<string> tokens;	size_t p0 = 0, p1 = string::npos;	while(p0 != string::npos)	{		p1 = str.find_first_of(delim, p0);		if(p1 != p0)		{			string token = str.substr(p0, p1 - p0);			tokens.push_back(token);		}		p0 = str.find_first_not_of(delim, p1);	}	return tokens;}int main(int argc, char * argv[]){	char baseDir[256] = {0};	GetCurrentDirectory(256, baseDir);	// Remove the patch file if it alreacy exists	std::string patchFileName = baseDir;	patchFileName += "\\patch.txt";	DeleteFileA(patchFileName.c_str());	// Try to download the patch file list	HRESULT result = URLDownloadToFileA(NULL, "http://members.gamedev.net/edxLabs/autopatch/patch.txt", patchFileName.c_str(), 0, NULL);	if(result != S_OK)	{		MessageBoxA(0, "Could not obtain the patch list.", "Fatal Error", MB_ICONERROR);		return 0;	}	std::ifstream inFile(patchFileName.c_str());	if(inFile.is_open() == false)	{		MessageBoxA(0, "Could not open the the patch.txt file.", "Fatal Error", MB_ICONERROR);		return 0;	}	// Create a tmp dir for files to download into	std::string tmpDirectory = baseDir;	tmpDirectory += "\\tmp";	CreateDirectory(tmpDirectory.c_str(), 0);	std::string buffer;	while(std::getline(inFile, buffer))	{		// Parse the input file		std::vector<std::string> parts = TokenizeString(buffer, "\t");		if(parts.size() != 3)		{			printf("The entry \"%s\" is invalid.\n", buffer.c_str());			continue;		}		std::string filename = parts[0];		std::string filecrc = parts[1];		std::string filepath = parts[2];		// Build a path toe the real file		std::string fpath = baseDir;		fpath += "\\";		fpath += filepath;		fpath += "\\";		fpath += filename;		bool doDownload = true;		// If the file exists, try to get the CRC		DWORD crc;		if(GetFileCrc(fpath.c_str(), crc) == true)		{			DWORD expectedCRC = HexStringToInteger(filecrc.c_str());			// If the CRCs match, we do not need to update			if(expectedCRC == crc)			{				doDownload = false;				printf("File %s does not need to be updated.\n", filename.c_str());			}		}		// If we need to download this file		if(doDownload)		{			// Build the path to the file to download			std::string baseUrl = "http://members.gamedev.net/edxLabs/autopatch/";			baseUrl += filename;			// Build a path to the tmp file we are about to download			std::string tmpFilename = tmpDirectory;			tmpFilename += "\\";			tmpFilename += filename;			// Try to download the file, this function blocks! You need to use the API more fully			// for better uses.			HRESULT result = URLDownloadToFileA(NULL, baseUrl.c_str(), tmpFilename.c_str(), 0, NULL);			if(result != S_OK)			{				printf("Could not download the updated %s file.\n", filename.c_str());				continue;			}			// We need to make sure the final file path exists, so we just recreate them			// for each file. This is not efficient for a lot of files, but this is			// just a simple example...			std::vector<std::string> paths = TokenizeString(filepath, "\\/");			for(size_t x = 0; x < paths.size(); ++x)			{				std::stringstream builder;				builder << baseDir << "\\";				for(size_t y = 0; y < x; ++y)				{					builder << paths[y] << "\\";				}				builder << paths[x];				CreateDirectory(builder.str().c_str(), 0);			}			// Remove the old file (should rename just in case the MoveFile fails!)			DeleteFileA(fpath.c_str());			// Try and copy over the new file			if(MoveFile(tmpFilename.c_str(), fpath.c_str()) == FALSE)			{				printf("Could not update: %s\n", filename.c_str());				// Remove the temp file				DeleteFile(tmpFilename.c_str());				// Try the next file				continue;			}			else			{				printf("Updating %s\n", filename.c_str());			}		}	}	inFile.close();	DeleteFileA(patchFileName.c_str());	RemoveDirectory(tmpDirectory.c_str());	return 0;}

This example makes no use of version numbers and is more along the lines of what you were asking about. If you want to try it yourself on your own server, all you need to do is create a "patch.txt" file that has a format of: filename<tab>crc<tab>destination path and then have the files in the same directory.

That's about as simple as you can get it (using Win32 at least). Just to reiterate the process is:
1. Client obtains a list of files
2. Client checks each file against its own
3. For each file in the list:
3a If the file is missing, download it to a tmp folder
3b. If the file exists and CRC matches, skip it
3c. If the file exists and CRC does not match, download it to a tmp folder
3d. Remove/Rename the original file and then replace it with the new file
4. Process is completed. If there were errors exit and tell the user to try again.

I'm suspecting your specific problems are on the Python side rather than the logic side. As mentioned before, can you give more information on the specific Python code or APIs you are using for the task as well as the web server information?

 Didn't see your post about the python code, but I'd not be much help suggesting alternatives in Python.

##### Share on other sites
I really appreciate the help. I actually was initially going to go the C++ route before, but I had issues with getting the particular download method to work. I will definitely look at your code and adapt to my needs.

Thanks again!

##### Share on other sites
Alright, this code has been EXTRAORDINARILY helpful, and I TRULY appreciate it.

My only concern is letter case. Is there any way to make the downloads case-insensitive without attempting many different versions of the file? (For instance, a lot of my files are .png, but some are .PNG, and if a .png file fails, I can simply retry as .PNG).

Thanks again!

##### Share on other sites
Quote:
 Original post by CrazyfoolMy only concern is letter case. Is there any way to make the downloads case-insensitive without attempting many different versions of the file? (For instance, a lot of my files are .png, but some are .PNG, and if a .png file fails, I can simply retry as .PNG).

On Windows, files are not case sensitive but on *nix they are. You might be able to work up a convoluted solution using apache and mod_rewrite or using your htaccess, but generally, if you are using *nix, you just have to make sure you set the names yourself correctly.

I'd just go to manually renaming all the files yourself to lower case or better yet, create a Windows program to do it and save yourself the headaches! For example, this is where a higher level language is very handy since you can just do something like this, a small VB.net program that you can compile and run in your file directory to easily rename the files' extensions to lower case.
Module Module1    Sub Main()        For Each foundFile As String In My.Computer.FileSystem.GetFiles(My.Computer.FileSystem.CurrentDirectory)            Dim strFile            Dim lastIndex            Dim tmpFile            Dim finalFile            Dim tmpParts            ' Breakup the file by path separators to get the file title            strFile = Split(foundFile, "\")            lastIndex = UBound(strFile)            finalFile = strFile(lastIndex).ToLower()            ' Get the extension for the title and make it lower case            tmpParts = Split(finalFile, ".")            lastIndex = UBound(tmpParts)            tmpParts(lastIndex) = tmpParts(lastIndex).ToLower()            ' Rebuild the string by adding back the .'s            finalFile = String.Join(".", tmpParts)            ' Store the temp name of the file since Windows needs             ' unique file names to rename properly            tmpFile = finalFile + "_"            Try                ' Rename the current file into the tmp file                My.Computer.FileSystem.RenameFile(foundFile, tmpFile)                ' Rename the tmp file into the final file                My.Computer.FileSystem.RenameFile(tmpFile, finalFile)                ' Status                System.Console.WriteLine("Successfully renamed the file " + finalFile)            Catch ex As Exception                ' Status                System.Console.WriteLine("Could not rename the file " + foundFile)            End Try        Next    End SubEnd Module

You could also take it one step further and just make it generate the patch.txt file for you automatically. In that case, you'd not even need to rename the files. You just would subtract out the base path from the current path to get the relative path, find a CRC32 function, and simply write out the final file.

I'd suggest an approach like that because you write the tool once, run it in your root patch directory, and just have to upload the file to your webserver. Heck, you can even just add the FTP code to the program itself and make it 100% automated. All you have to do is run the exe [smile]

So, there's lots of cool things to do to make life easier, I'd defeintly suggest the route of making your own small utility in a language of your choice for it. Whatever makes the task as easy as possible. I'm not even a VB.net person but you can see how simple .net makes things such as this, so choosing the right tools for the job makes all this stuff more fun!

Just some ideas, good luck!

##### Share on other sites
Thanks a bundle. I think this has been the 100th time you've help me.

Edit: And I agree about the higher level language. While my solution is going to be in C++ (thanks to you), playing around in Python was quite fun and enjoyable. It also taught me some things, and I think I could have easily done it with Python had I stuck with it (Compare my 1-2 days of Python experience and being able to get something to at least marginally work to the years of C++ experience and having difficulties ;) )

1. 1
Rutin
24
2. 2
3. 3
JoeJ
18
4. 4
5. 5

• 38
• 23
• 13
• 13
• 17
• ### Forum Statistics

• Total Topics
631710
• Total Posts
3001845
×