• entries
35
32
• views
26957

## The worst PC problem I've *ever* had...

The past couple of months I've had a really nasty PC bug plaguing my productivity. It all started when one of my development PCs (XP SP3) would randomly freeze in the middle of me doing programming stuff. At first I thought it was just a fluke since it happened so infrequently but eventually the problem intensified. Since I had two development machines, both with the same hardware, I just switched over to the second one and used the first one less frequently. That's why I built two low budget development machines instead of a more expensive higher end one. In the case something should happen to one of them, I'd always have the other to fall back on.

Anyways, everything seemed to be fine on the second PC for quite some time until one day it froze just like the other one.

O-M-G

For the next month or so, I just tried to work around the problem by saving often, but then the problems seemed to intensify more and more. Now, I had two almost identical machines with the same problem. At first, I thought it was because of my hard drives. They both seemed to report some S.M.A.R.T. issues so I went ahead and swapped out the hard drives in one machine with some other spare ones. I spent a couple of days reinstalling everything and getting my development environment back up. It seemed ok at first, but after starting up Visual Studio 2008, *BAM*, freeze.

So now I know it wasn't the hard drive. Next, I figured it was possibly the ram. I've never had ram go back on me before, but I went ahead and grabbed MemTest and ran it for a day to see if that was the cause. Almost 24 hours of running it and 0 errors. It wasn't the memory either...

At this point, I'm really aggravated since I can't get anything done since the freeze only happens with Visual Studio and I spend the amjority of my day working in Visual Studio. I never had the freeze trigger from movies or games or music, so it was really becoming a nightmare.

I decided to check the video card next. I was using an old ATI 3870, but I didn't really think it would be the problem since playing intensive video games never locked up the PC. I took it out and enabled my onboard graphics, a HD 4200 chipset. Lo' and behold, I start up Visual Studio and it freezes again!

RAGE!

This time though, I got a nice BSOD after the freeze referencing the ATI video driver. I'd never gotten that error before but it mentioned an endless loop error code. I had not removed my ATI Catalyst drivers before trying the integrated graphics so I went ahead and uninstalled it. After I rebooted, I installed the default drivers for the integrated graphics and attempted to start Visual Studio 2008.

No freeze!

Following the next logical step in this process, I reinstalled my 3870 but choose an older 9.X version of the Catalyst drivers. I started up Visual Studio 2008 once again and got no freezes! I then uninstalled the 10.X drivers on my other development PC and installed the 9.X ones and retested the same way.

Yay, problem solved! (for now, I think)

I still have to do more time consuming regular use testing, but at least now it seems the problem has been fixed. I simply can't believe running the latest AMD drivers caused such a horrible problem like this. I thought it was software related at first, but I never thought about graphics drivers being the culprit. I use Visual Assist X so I was beginning to think it was the problem until I had uninstalled it and still got the same freezes. I also tried different anti-virus and other software I thought might have done it, but I always got the same freeze each time.

This has been by far the worst PC problem I've ever had since you have to litterally check *everything* hardware and software just to try and track down the problem. At least now I know to check for downgrades first before going crazy reinstalling everything over and over and messing with hardware.

## Development Update!

I'm project jumping again, but this time I think I'm back on the right path.

I got my 360 and looked into some C# and XNA development and I just couldn't get into it once again. I don't know, maybe it'll take another year or so but I'm stuck in the C/C++ level. I've been playing some Xbox 360 games and don't think I could even compete on that level for middleware or anything for indies. Back to the PC I go for now and will just have that as my leisure activity.

Back in March, I made an entry on a game engine project I wanted to start on. Long story short, that went down the drain fast after seeing how Newton was setup and the work required to start porting it into the engine. I got discourage and decided to spend more time on playing with the Leadwerks engine instead, which I had started with in January.

Well after spending 6 months with the engine, I've gotten some good ideas of how I should structure my engine. The Leadwerks engine is a real example that shows how engines can be simple yet efficient and powerful still. There are a few drawbacks to the engine itself though, since it is based in Blitzmax, but overall it is a very nice engine. It's one I would still recommend for people to play with and one I'd still support.

Anyways though, I'm starting my own engine back up but this time I have a more clear path I want to take. This time around, I will be using:
* SFML for the application framework, audio, input, etc...
* Lua for scripting interactions
* OpenGL for graphics rendering, shaders, etc..
* PhysX for physics
* Xinput for my secondary input
* Configurable Math Library for additional math related stuff
* My UDP IOCP framework code alongside enet for networking
* CEGUI for game gui.
* Other mini code examples and libraries where needed

The deployment strategy I will be using is the one that the Leadwerks engine uses as well as another engine I can't recall the name of right now. The way it works is through a DLL so any language that supports C strings and loading DLLs can use the engine. It's a great strategy if I might say.

Basically, I'm just combining everything and writing the glue code to use the components via my own API and then the appropriate Lua binding glue code to expose those to the scripting side. It's a bit of work, but it's what I want to do and I'm working hard each day to make more progress and get something working. The way I am starting out is just getting everything working first, and then I'll go back and add more flexibility, since right now I've got a few things limited in terms of only being able to have one instance at a time.

Once that grunt work is done, I have to look into some model formats and integrating simple shader code so things just don't look flaccid. However, I'm not going to focus on the graphics side of things that much. My main goal is getting everything else working together to be able to prototype out a network based game. This stage is a bit away, but it's on the radar.

So, that's it for my little development update. I don't think this project is going to get abandoned like my previous ones. I really don't have anything else to work on for the time being, so I think it's a good time to get this done and start making the magic happen! I'm not sure when I'll post another update, but hopefully in a week or two with a lot more progress done.

## Star Wars Force Unleashed Disappointement

My Xbox360 came yesterday and I got the first game I ordered as well. I remember having watched the game's trailer when it was first announced and thinking how I really wanted to play it. Well, After spending less than half a day playing through it, I've beat the game and completed the main story. I am sourly disappointed in how short the game was. I must have played less than 12 hours total or so. It was a lot of fun, but it was a bit just too short.

There is still more to do, achievement wise with the game, but now that the story is over, I'm a bit meh. The story itself was somewhat enjoyable, but I didn't like the ending. I won't spoil anything for anyone who is going to look into it, but once again the key word is disappointment.

I paid a little over 50 or so for the game. I could not recommended it at that price. Perhaps 30 or the 40 range if you are a die hard Star Wars fan. The game itself has quite a few nasty bugs due to physics. A lot of times I died in levels without even knowing what to do and it was greatly discouraging dying over and over until I figured out what to do.

That concludes my short and initial review of the game. On a scale of 1 to 10 I would give the game a 6. The story and gameplay is linear with no decisions, figuring out what to do in numerous spots in the game make it frustrating at times, and the lurking physics bugs and save system can further add to displeasure if you get stuck and have to restart a ways back.

## Xbox 360 Development

I've decided to take a stab at Xbox360 + Windows game development through XNA. I find myself spinning my wheels yet again with a lack of motivation and inability to materialize a C++ based project. I have ideas of frameworks I want to development, but I just can't seem to get it going. I think I'm at a point of being burned out from PC development for the time being.

I've been keeping an eye on this Hanns.G HG-281DPB Black 27.5" monitor for a while now. Thinking how good of a deal it was and I finally pulled the trigger on it tonight after seeing the additional instant rebates. I was thinking it was a good deal at $350 a day ago but for$310 today, it's a real steal I couldn't pass up any longer. I was considering a 42" 1080P LCD from Westinghouse, which was about $580 at Tiger Direct, but I'm just not a big TV person. I have a 32" LCD 720P from the family I use to watch my daily news with, but aside from that I just don't like laying down and watching TV. I much prefer being at my desk doing stuff on my computer, so I'll be able to have the monitor there and hook up the 360 and keep it close by. I have a nice recliner near my desk I can use for gaming as well, which would be nice. Come to think about it, I need to get another long Ethernet cable since my router is on the other side of the room. Darn. Actually it might be better just to get a switch for my desk and then run the desktop and 360 from there to the router. Oh well, still plenty of time to order before Monday. I usually over think all my purchasing decisions and 99.9% of the time decide to pass, so this time I just ordered and will not look back. Hopefully this gaming break on a console and then shifted development to XNA and C# will help me a lot. I'm really struggling here with a lack of focus and it's killing me. I need to enjoy myself for once, which lately I have been doing in the form of playing Final Fantasy 7 on PC. I've actually almost gotten to where I was when I had last played it on PS1 many many years ago in about 1/2 the time. It sucks though because the window resolution is a crappy 640 x 480. I'd love to make my own FF7ish game, that's definitely on the to-do list, along with bomberman, a gta like clone, you know, the good stuff. Maybe I should just play games and not worry about trying to make one or tools for them anymore. Doing this stuff on your own is hard, so I have a lot of respect for all the indies out there that are doing it, regardless of how successful they are or not. In theory, I should be more than capable of making a game, but something is holding me back and I don't know what yet. Anyways, that's about it for now. Hopefully I can get something going in the upcoming months playing around with this new stuff. I do plan to spend a couple of weeks zoning out on games to clean out the system, but after that more hard work! ## Win7 = Thumbs Up! So I got my free Win7 copy this morning and installed it earlier. I've been using it all day and so far I like it. I do have to get used to the taskbar though, but aside from that I like the overall speed and feel of it. So far I've been able to get drivers for hardware fast and easily with Windows Update, so that's definitely a plus. I do have an issue with not being able to get 5.1 sound and only 2.1, but I can live with that for now. I loaded up Visual Studio 2008 and have been doing some updates on other code and haven't had any hiccups or anything. At this rate, unless something just startling pops up anytime soon, I plan on building a new PC end of the year and use Win7 as the future OS to do my work in. I still have a few more programs to install and evaluate, but so far so good. I'm happy as a clam. ## May is barreling down on me And I've yet to think about what to work on. I've been too busy following news and the stock market in detail. Could have had a nice little return yesterday, but waited too long and ended up with a small return. I sat through a 4 hour audio case of a Bank of America shareholder meeting. ZZZzzzZZZ, I can't believe I listened to it all. Quite a lot of angry folks, just goes to show why Buy and Hold is a bad strategy. I created a Twitter account, but I'm not sure when I'll start using it. I really need a new phone too, mines from like 2000 and I barely use it. I need to get a new modern one this year, I think. I'm looking forward to seeing Wolverine soon, I got a free ticket from buying the XMen trilogy Blu-Ray bundle. I can't wait for the summer hits to come, there's some good ones I can't wait to see. What else is there. I've read one of the four books I got to read on the stock market. I picked up 4 out of 5 books Jim Cramer has written. It's been a while since I've read a non-technical book. I need to catch up on the Colbert Report and Daily Show, have a few episodes. Likewise I need to listen to the Jon Stewart audio book. The Colbert audio book had some good laughs in it. That about wraps up this non-development post update. Not much going on really. ## Reflections on 1Q 2009 With the second quarter already underway in 2009, I wanted to go ahead and post some reflections of how this year has gone so far. Lately, I've been reading up on using physics over a network and synchronization and the issues to work around and I've been slightly put off. It seems there is no generic way to handle this and the path you take is highly dependent on your game. Since I don't actually have a game, that puts me in a little bit of a bind. I wanted to try to use Havok physics in my project, but I really don't feel like learning it with no specific task at hand. I certainly love all the features it has and the demos are amazing, but for some reason I don't see the utility in it right now. Newton physics was just a mess project wise and now I am thinking I should not even have a base physics system in my project. I think the real problem is I don't have a focus at the moment. I have all these things I want to do, but no concrete ultimate goal. 2 years ago, I wanted to learn reverse engineering and low level programming and spent 2 years getting good at it. A year ago, I wanted to learn how to do network coding so I spent a year working with TCP. 4 months ago, I wanted a better networking system, so I spent the time working on one. All of that is done done now and I did learn a lot, so it's on to the next thing. What's the next thing though? * I have no knowledge of graphics related stuff, like rendering, shaders, and everything in that domain. I mean I could spend a year or so on that, but I've never been a big graphics person from a programming perspective. * I do have a todo entry for learning device drivers and the like on Windows, but I see that becoming less relevant for me as Windows approaches the post XP era. I don't think I'll even have a real need for this now. * I'm not up to date on the latest web technologies. Then again, I could never do web design, only web programming, so learning ASP.Net would be the key thing here. I do want to make more net driven applications that do not solely revolve around an actual program server but rather a web server, so I do have some interest here. * I need to learn more programming languages that you don't use the monolithic design principle with as you do with C/C++. I wanted to learn erlang a while ago and Ada, but I just couldn't commit to it. Even my advances with Lua slowed down, but I did have some good use out of it for a time. I think that sums up general domains of interest I am lacking in skill wise at the moment. I think I will be taking it easy the next week or so and try to find something to concentrate on for the month of May. I think I am just getting burned out again as this feeling has been happening more frequently as the years pass. I now see the benefits of having a job and much less free time. I'm not complaining though, I don't mind my current situation. However, not talking or interacting with people on a daily basics and not having any real "friends" is starting to take its toll. QQ for me, right? Anyways, maybe I should spend less time programming and start writing again. I still need to write my own book and that would certainly take up a bit of time I find myself having between regular programming. I've also taken an interest in learning how stocks are traded and have been looking to try and make Mad Money. Not for the money though, for the excitement and experience. Trying to get good at something is always a challenge I like to take, and this stuff is outside the regular norm of things I concentrate on. Decisions, decisions. I ordered a bunch of stuff from Amazon yesterday as I've been wanting a few things and just been holding out on buying anything for no good reason. I saw a watch on Amazon in Match I really liked, but it wasn't released until a week ago or so, so I got that. I ordered some of my favorite Old Wisconsin turkey bite snacks because those things rock but are often out of stock. Finally, I got a bunch of blu-ray movies for me and the family since I think they are worth it over the DVD versions we have of them. I've also been thinking a lot about getting additional computer hardware, but I can't seem to justify a reason for it. I've spent literally hundreds of hours researching servers, SFF builds, and looking at new technology coming out and just can't find a reason for wanting to actually upgrade anything. I think I will upgrade my video card when Win7 comes out, but by then I will just build a new computer as I don't want to lose this one due to having so many node-locked licenses on here. Looking forward, I don't quite know what I will be aiming for. i have been trying to horde up experience and that's been going well, but as I mentioned above, I am kind of running out of basics to learn, which is a good thing, but leaves the question of what to do afterwords. I'm sure I'll think of something though, I've certainly got the time right now and the ability, just finding the right logic based motivation can be hard. I think that's all for now, I'm going to listen to I Am America (And So Can You) on my IPod since I just got the audio book the other day on ITunes. I've been watching The Colbert Report and The Daily Show a lot lately and loving it. I think The Daily Show is more of my favorite though. Good stuff [smile] ## 4/17/09 Status Report: Intergration Done! If you look at the previous status report, you will notice that the test client code I showed was a little messy. The reason was because I was using both the modified enet and my UDP IOCP code at the same time without creating an interface between them. Today, I combined my progress of enet along with my IOCP code and wrapped it into a simple and easy to use library. It works great! I think it is very simple to use overall and very efficient. Here is the new sample code for the server and client. Server #include "edxnet.h" #include //----------------------------------------------------------------------------- // We will use this event as a signal to exit. We simply block the main function // and let the server do all the work in its own threads. This won't be needed // if you have a main loop to execute. HANDLE hExitEvent = 0; //----------------------------------------------------------------------------- // We can use a console handler to detect console closing events to gracefully // handle when the user wishes to exit. BOOL __stdcall ConsoleHandler(DWORD ConsoleEvent) { switch(ConsoleEvent) { case CTRL_LOGOFF_EVENT: case CTRL_C_EVENT: case CTRL_BREAK_EVENT: case CTRL_CLOSE_EVENT: case CTRL_SHUTDOWN_EVENT: { SetEvent(hExitEvent); return TRUE; } } return FALSE; } //----------------------------------------------------------------------------- // Console entry point to save us from having to write a GUI int main(int argc, char * argv[]) { // Our UDP host EdxNetHost host; // This main thread, which we want to process packets in the context of HANDLE hMainThread = OpenThread(THREAD_GET_CONTEXT | THREAD_SET_CONTEXT, FALSE, EdxNet_GetThreadId()); if(hMainThread == NULL) { printf("Failed to OpenThread: %i\n", GetLastError()); system("pause"); return 0; } // Setup the library, this call will setup the UDP library as well if(EdxNet_Initialize() == false) { printf("Error: InitializeEdxNet failed.\n"); return 0; } // Create our UDP host if(host.Create(hMainThread, 15779, 1024, 0, 0, 1, 1, 0, 0) == false) { EdxNet_Deinitialize(); printf("Error: Could not create the EdxNetHost.\n"); return 0; } // So we know when to exit hExitEvent = CreateEvent(0, TRUE, FALSE, 0); // Set a console handler to make our program more flexible and simple. We will be able to // Ctrl + C out of our program rather than have to code in a more complicated GUI. SetConsoleCtrlHandler(ConsoleHandler, TRUE); // Loop until we need to exit while(WaitForSingleObjectEx(hExitEvent, 1, TRUE) != WAIT_OBJECT_0) { int result; EdxNetEvent event; while((result = host.Service(event)) == 1) { if(event.type == EDXNET_EVENT_TYPE_RECEIVE) { __int64 recvIndex = event.packet->Read(); EdxNetPacket * packet = EdxNetPacket::Allocate(); packet->SetFlags(true, true); packet->Append(recvIndex); // Send the packet to the peer, we do not need to free // the object ourself. event.peer->Send(0, packet); // We will need to deallocate the packet of the event // to keep memory usage as optimized as possible. EdxNetPacket::Deallocate(event.packet); } else if(event.type == EDXNET_EVENT_TYPE_CONNECT) { printf("Client [%X] has connected.\n", event.peer); } else if(event.type == EDXNET_EVENT_TYPE_DISCONNECT) { printf("Client [%X] has disconnected.\n", event.peer); } } } // Cleanup resources CloseHandle(hMainThread); // Remove the console handler as we no longer need it. SetConsoleCtrlHandler(ConsoleHandler, FALSE); // Cleanup the host host.Destroy(); // Finally cleanup the library EdxNet_Deinitialize(); // Standard return return 0; } //----------------------------------------------------------------------------- Client #include "edxnet.h" #include //----------------------------------------------------------------------------- // We will use this event as a signal to exit. We simply block the main function // and let the server do all the work in its own threads. This won't be needed // if you have a main loop to execute. HANDLE hExitEvent = 0; //----------------------------------------------------------------------------- // We can use a console handler to detect console closing events to gracefully // handle when the user wishes to exit. BOOL __stdcall ConsoleHandler(DWORD ConsoleEvent) { switch(ConsoleEvent) { case CTRL_LOGOFF_EVENT: case CTRL_C_EVENT: case CTRL_BREAK_EVENT: case CTRL_CLOSE_EVENT: case CTRL_SHUTDOWN_EVENT: { SetEvent(hExitEvent); return TRUE; } } return FALSE; } //----------------------------------------------------------------------------- // Console entry point to save us from having to write a GUI int main(int argc, char * argv[]) { // Our UDP host EdxNetHost host; // This main thread, which we want to process packets in the context of HANDLE hMainThread = OpenThread(THREAD_GET_CONTEXT | THREAD_SET_CONTEXT, FALSE, EdxNet_GetThreadId()); if(hMainThread == NULL) { printf("Failed to OpenThread: %i\n", GetLastError()); return 0; } // Setup the library, this call will setup the UDP library as well if(EdxNet_Initialize() == false) { printf("Error: InitializeEdxNet failed.\n"); return 0; } // Create our UDP host if(host.Create(hMainThread, 0, 0, 0, 0, 1, 1, 32, 16) == false) { EdxNet_Deinitialize(); printf("Error: Could not create the EdxNetHost.\n"); return 0; } // Connect to the server EdxNetPeer * peer = host.ConnectTo("192.168.1.121", 15779, 1); // So we know when to exit hExitEvent = CreateEvent(0, TRUE, FALSE, 0); // Set a console handler to make our program more flexible and simple. We will be able to // Ctrl + C out of our program rather than have to code in a more complicated GUI. SetConsoleCtrlHandler(ConsoleHandler, TRUE); // bool bCanSend = false; // __int64 realIndex = 0; // double lastSendTick = 0; // Loop until we need to exit while(WaitForSingleObjectEx(hExitEvent, 1, TRUE) != WAIT_OBJECT_0) { int result; EdxNetEvent event; while((result = host.Service(event)) == 1) { if(event.type == EDXNET_EVENT_TYPE_RECEIVE) { __int64 recvIndex = event.packet->Read(); if(recvIndex != realIndex) { printf("Uh-oh, data mismatch! %lld vs %lld\n", recvIndex, realIndex); SetEvent(hExitEvent); break; } double latency = EdxNet_GetTickCount() - lastSendTick; printf("[%lld] %.12f\n", recvIndex, latency); // We will need to deallocate the packet of the event // to keep memory usage as optimized as possible. EdxNetPacket::Deallocate(event.packet); bCanSend = true; } else if(event.type == EDXNET_EVENT_TYPE_CONNECT) { printf("Client [%X] has connected.\n", event.peer); bCanSend = true; } else if(event.type == EDXNET_EVENT_TYPE_DISCONNECT) { printf("Client [%X] has disconnected.\n", event.peer); SetEvent(hExitEvent); } } if(bCanSend && EdxNet_GetTickCount() - lastSendTick >= 50) { EdxNetPacket * packet = EdxNetPacket::Allocate(); packet->SetFlags(true, true); packet->Append(++realIndex); packet->Append(0); packet->Append(0); packet->Append(0); // Send the packet to the peer, we do not need to free // the object ourself. peer->Send(0, packet); bCanSend = false; lastSendTick = EdxNet_GetTickCount(); } } // Disconnect from the server peer->Disconnect(); // We need to flush the host so the disconnection request is // actually sent out. If we want a quick exit, we can just call // this and let the server timeout the client if the packet is // lost. //host.Flush(); // If we want a graceful exit, we can wait up to 5 seconds for // the confirmation packet, and then just shutdown if it was // not received. bool bExit = false; double startTick = EdxNet_GetTickCount(); while(EdxNet_GetTickCount() - startTick 5000 && bExit == false) { int result; EdxNetEvent event; while((result = host.Service(event)) == 1) { if(event.type == EDXNET_EVENT_TYPE_DISCONNECT) { printf("Client has successfully disconnected.\n", event.peer); bExit = true; break; } } if(bExit == false) { SleepEx(1000, TRUE); } } // Cleanup resources CloseHandle(hMainThread); // Remove the console handler as we no longer need it. SetConsoleCtrlHandler(ConsoleHandler, FALSE); // Cleanup the host host.Destroy(); // Finally cleanup the library EdxNet_Deinitialize(); // Standard return return 0; } //----------------------------------------------------------------------------- All of the server configuration is done through the Create function call of the host. Since I liked the enet way of doing things, and that is using host/peer rather than server/client, the server networking code is the same as the client networking code! In the past, I made separate client and server classes to handle events, but this model works great for that in clients can harness the power of a normal server as well. I did not quite finish all of the final enet changes I had wanted to, but I can go back later on improve on those. I did not want to spend more time on those things and still have to work through the integration part, so I decided to 'move ahead'. It's definitely not in a release state yet, but it's getting there. I'll go ahead and paste the current user API, but as you will see, it's not finished yet in terms of comments. I do need to add a few more utility functions such as the ability to get the current port number used. That will make P2P possible using this setup as right now you have to know the port you are connecting to! #ifndef EDXNET_H_ #define EDXNET_H_ //----------------------------------------------------------------------------- #include //----------------------------------------------------------------------------- // How many bytes the internal buffer size is const int HPS_BUFFER_SIZE = 8192; //----------------------------------------------------------------------------- // We can use these macros to easily convert between host and // network byte order #define EDXNET_HOST_TO_NET_16(value) (htons(value)) #define EDXNET_HOST_TO_NET_32(value) (htonl(value)) #define EDXNET_NET_TO_HOST_16(value) (ntohs(value)) #define EDXNET_NET_TO_HOST_32(value) (ntohl(value)) //----------------------------------------------------------------------------- // Forward declarations struct EdxNetPacketData; class EdxNetPacket { public: // PIMPL object to keep the interface clean EdxNetPacketData * internalData; private: // Since we are using templates, we can't hide these away :( UINT8 * workspace; int writeIndex; int readIndex; bool wasError; // This object is a special type, we need to use // a special allocator for the object. As a result, // we will not be allowed to use these operations // on the type. private: EdxNetPacket(); ~EdxNetPacket(); EdxNetPacket(const EdxNetPacket & rhs); EdxNetPacket & operator =(const EdxNetPacket & rhs); // Update internal data members. void Finalize(); public: // Allocates the packet object and required memory static EdxNetPacket * Allocate(); // Deallocates the packet object and all associated memory static void Deallocate(EdxNetPacket * packet); // Set the packet flags as needed void SetFlags(bool reliable, bool sequenced); // Returns a pointer to the packets data UINT8 * GetDataPtr(); // Returns the data's number of bytes size_t GetDataSize(); // Seek the read or write index or obtain the current // position of the index. size_t Seek(UINT8 mode, int count, bool set); template class type> void Append(type t) { size_t sizeoft = sizeof(t); if(writeIndex + sizeoft >= HPS_BUFFER_SIZE) { wasError = true; return; } memcpy(workspace + writeIndex, &t, sizeoft); writeIndex += sizeoft; Finalize(); } template class type> void AppendArray(type * t, size_t count) { size_t sizeoft = sizeof(t[0]) * count; if(writeIndex + sizeoft >= HPS_BUFFER_SIZE) { wasError = true; return; } memcpy(workspace + writeIndex, t, sizeoft); writeIndex += sizeoft; Finalize(); } template class type> void Overwrite(int position, type t) { size_t sizeoft = sizeof(t); if(position + sizeoft >= HPS_BUFFER_SIZE) { wasError = true; return; } memcpy(workspace + position, &t, sizeoft); if(position + sizeoft > writeIndex) { writeIndex = position + sizeoft; Finalize(); } } template class type> void OverwriteArray(int position, type * t, size_t count) { size_t sizeoft = sizeof(t[0]) * count; if(position + sizeoft >= HPS_BUFFER_SIZE) { wasError = true; return; } memcpy(workspace + position, t, sizeoft); if(position + sizeoft > writeIndex) { writeIndex = position + sizeoft; Finalize(); } } template class type> type Read() { type t = type(); int s = sizeof(type); if(readIndex + s > HPS_BUFFER_SIZE) { wasError = true; return t; } memcpy(&t, workspace + readIndex, s); readIndex += s; return t; } template class type> void ReadArray(type * array, unsigned long count) { int s = sizeof(type) * count; if(readIndex + s > HPS_BUFFER_SIZE) { wasError = true; return; } memcpy(array, workspace + readIndex, s); readIndex += s; } }; //----------------------------------------------------------------------------- // Forward declarations struct EdxNetPeerData; // class EdxNetPeer { public: // PIMPL object to keep the interface clean EdxNetPeerData * internalData; public: // The ctor will setup the internalData object EdxNetPeer(); // The dtor will cleanup the internalData object ~EdxNetPeer(); // Sends the passed packet on the specified channel. Returns 0 on success // and -1 on error. int Send(UINT8 channelID, EdxNetPacket * packet); // Disconnects this peer. if mode is 1, the disconnect is gradual as soon // as the peer has no more data to send. As soon as no more data is queued, // the default disconnect process is used. If mode is 2, the disconnect is // instant and forced. If mode is anything else, the default disconnect // process is used. void Disconnect(UINT8 mode = 0, UINT32 data = 0); }; //----------------------------------------------------------------------------- enum EdxNetEventType { EDXNET_EVENT_TYPE_NONE = 0, EDXNET_EVENT_TYPE_CONNECT = 1, EDXNET_EVENT_TYPE_DISCONNECT = 2, EDXNET_EVENT_TYPE_RECEIVE = 3 }; struct EdxNetEvent { EdxNetEventType type; EdxNetPeer * peer; UINT8 channelID; UINT32 data; EdxNetPacket * packet; EdxNetEvent(); }; //----------------------------------------------------------------------------- // Forward declarations struct EdxNetHostData; class UDPHighPerformanceServer; struct sockaddr_in; // class EdxNetHost { public: // PIMPL object to keep the interface clean EdxNetHostData * internalData; public: // The ctor will setup the internalData object EdxNetHost(); // The dtor will cleanup the internalData object ~EdxNetHost(); // bool Create(HANDLE hThread, UINT16 port, size_t peerCount = 32, UINT32 incomingBandwidth = 0, UINT32 outgoingBandwidth = 0, UINT32 numberOfConcurrentThreads = 1, UINT32 numberOfWorkerThreads = 1, LONG maxPendingRecvs = 0, LONG refillPendingRecvs = 0); // void Destroy(); // EdxNetPeer * ConnectTo(const char * name, UINT16 port, size_t channelCount); // int DeqeueEvent(EdxNetEvent & eevent); // void Flush(); // void LimitBandwidth(UINT32 incomingBandwidth, UINT32 outgoingBandwidth); // void ThrottleBandwidth(); // int Service(EdxNetEvent & eevent); // void Broadcast(UINT8 channelID, EdxNetPacket * packet); // void PostIncomingCommand(sockaddr_in * address, UINT16 count, UINT8 * data); }; //----------------------------------------------------------------------------- // bool EdxNet_Initialize(); // void EdxNet_Deinitialize(); // DWORD EdxNet_GetThreadId(); // double EdxNet_GetTickCount(); // double EdxNet_GetTime(bool restart = false); //----------------------------------------------------------------------------- #endif So, I'm at the last lap of this current project. I still have about 2 weeks before the end of April, so I should definitely be "done" before the end of the month. I probably will want to put together a few demos using the code before a release anyways. I won't even be able to guarantee that this model of networking code will work better than the original enet code performance wise, but that's not important to me right now. What I do care about was the learning process of IOCP and UDP, which I've learned quite a lot. Sadly, in lieu of Raknet becoming free for Indies to use, I don't foresee much need for this project in the real world besides just for study. My base UDP IOCP code might prove useful for Windows devs who haven't wrapped their head around the concept, but I don't see the whole enet integration being popular. I have changed the enet code a little enough to where most people might not even want to bother, but the project is more aimed at a tech demo. The work I've spent in making enet compatible was solely to show off Stage 1, which is the real gem here. Anyways though, I'm one happy camper. I've enjoyed the process and hope to improve upon it more. I will probably won't have any new news for this prject until release date sometime in the next two weeks or so, so my next posts will probably be over physics or my other engine related works; I'm not sure yet. I would like to make a demo using the Leadwerks engine and my networking code though, so maybe I'll do that next. Thanks for reading! ## Havok Physics All I can say is wow! I will be giving it a whirl for use in my engine rather than messing with Newton Dynamics for now. I know I have to wrap up the interface since I can't link to it directly in the middleware, but I was very impressed looking through the demos. The concept of the phantoms is marvelous, the MOPP system looks great, and the whole threading system for more realistic/powerful dynamics scales well. I'm so looking forward to this [smile] ## 4/16/09 Status Report: Victory is near! This whole week I've been hard at work updating the code of enet and modifying it so I can use my IOCP UDP code alongside it. It's been going great overall and I'm getting closer to the finish line. Here's a quick task list of what I am working on right now with the code: * Get the last malloc use removed and implement a pool for the channel objects. * Remove the custom C style list and substitute in the std::list (might actually leave this alone) * Continue code cleanup for consistency and ease of reading * Look for additional optimizations in the most frequently used functions * Rename a lot of objects to reflect their new purpose. I've updated all the functions to modern C++ styled names, but the exposed objects need a few fixings. As of right now, the code really does work and I have a nice interface for it. I've integrated my old stream building and reading template based operation classes into the packet class to allow for a nice and simple way to build and read packets. Unfortunately, since it's template based, the header file is not as clean and tidy as I was aiming, but I can accept that trade off. I also identified and partially fixed a vulnerability in the enet code that has been mentioned and that involves the auto fragment system. A malicious user can send a packet to the server that is "very large" and the system will allocate memory for it. The problem is if a user were to send let's say 10 packets each 4gb in size, then you can see the problem, the server would instantly crash since a malloc will fail and the enet code is set to abort on that case. What I did was check the fragment count and only allow 32 total fragments. In essence this means only 44.8kb could be sent at a time, but even that has a vulnerability problem. So what I did was limit all packets to a maximum size of 8kb, or well 4kb probably in the final version, so those packets can be fragmented, but no dynamic memory allocations are being made by the system. I use boost singleton pool to manage all my memory, so allocations and deallocations are fast, efficient, and almost free since no dynamic memory is actually being created and destroyer in real time. Likewise, the other internal data types enet uses I've witched over to the pool system to help stabilize the library for servers that will be running for a long time. I've only got one more object to do and then that system is done! The packet system uses a pool as well, which in my high performance networking code, is absolutely vital! I love the PIMPL pattern and have made judicious use of it in enet now. Only a few member functions and data is actually exposed to the end user. Of course, anyone that wants to modify the underlying library settings will do so, but end users of the library will not need to. I'd like to learn more of the underlying algorithms and constants used, but it's quite overwhelming and takes a lot of time to understand correctly. Finally, that leads me to a recent test of the application using VTune. I ran about 100 clients from my vista laptop and hit a network card barrier due to the traffic being short circuited directly to the destination computer on the LAN. Each client sent 8kb each 100ms and the data was as being Reliable. I maxed traffic out around 10 megabytes per second. I'd liked to have done more, but I need to find why my link is only capping at 10 when it's a 10/100/1000 card I think. Looking over the results of more realistic tests, 1024 clients sending 1024 bytes a second, most of the time spent is in the kernel, namely in the IO completion port API functions. Here's some loose results, all times are in microseconds, not milliseconds! As you can see, about 1/2 the calls made are in NTDLL from the IOPC API and 96% of the time was spent in that module in test 3 and test 2 had 97% time spent. Test 2 had one worker thread and one thread allowed in the IOCP and test 3 had 8 worker threads and 4 threads allowed in the IOCP, These tests are by no means "scientific". Overall though, I am very pleased with the performance and results. It is more than suitable for a game since I am testing the worse case scenario with all reliable packets on a lan link with very low latency. I've also done some packet sending and receiving verifications tests to make sure the reliable logic works right and so far it still does. One thing to understand is that it's quite easy to break something accidentally through the code cleanup due to how the original code was written. I've have to be careful in that regard. I am glad I went ahead and started stage 2 before releasing stage 1. I've got a few interface changes to make to allow sending multiple buffers at once. I did not have that originally, but enet needed it to work properly, so it is now a part of the interface. Here is the main testing program I am using right now. This is pretty much the basic structure an application will follow if it wants to use enet alongside the library. I use the APC API again to make sure I get a thread safe implementation that only executes in one thread's context. I got a little clever in being able to keep that thread context the main thread, which is perfect for my needs! #include "UDPHighPerformanceServer.h" #include "edxnet.h" #include //----------------------------------------------------------------------------- // We will use this event as a signal to exit. We simply block the main function // and let the server do all the work in its own threads. This won't be needed // if you have a main loop to execute. HANDLE hExitEvent = 0; // Main thread handle HANDLE hMainThread = 0; //----------------------------------------------------------------------------- // We can use a console handler to detect console closing events to gracefully // handle when the user wishes to exit. BOOL __stdcall ConsoleHandler(DWORD ConsoleEvent) { switch(ConsoleEvent) { case CTRL_LOGOFF_EVENT: case CTRL_C_EVENT: case CTRL_BREAK_EVENT: case CTRL_CLOSE_EVENT: case CTRL_SHUTDOWN_EVENT: { SetEvent(hExitEvent); return TRUE; } } return FALSE; } //----------------------------------------------------------------------------- struct tOnClientToServerEvent { UDPHighPerformanceServer * server; sockaddr_in address; UINT16 count; UINT8 data[EdxNetPacket::TypeSize]; }; //----------------------------------------------------------------------------- typedef boost::singleton_poolsizeof(tOnClientToServerEvent)> Pool_Events; //----------------------------------------------------------------------------- VOID CALLBACK EnetProcessEventWrapper(ULONG_PTR param) { tOnClientToServerEvent * apcEvent = reinterpret_cast(param); EdxNetHost * host = reinterpret_cast(apcEvent->server->GetUserData()); host->PostIncomingCommand(&apcEvent->address, apcEvent->count, apcEvent->data); Pool_Events::free(apcEvent); } //----------------------------------------------------------------------------- // Our packet processing function. We have access to everything we need to handle // packets. This function will be thread safe by internal design for each // individual server. void WINAPI OnClientToServer(UDPHighPerformanceServer * server, sockaddr_in & address, UINT16 count, UINT8 * data) { tOnClientToServerEvent * event = reinterpret_cast(Pool_Events::malloc()); event->server = server; event->count = count; memcpy(&event->address, &address, sizeof(sockaddr_in)); memcpy(event->data, data, count); if(QueueUserAPC(EnetProcessEventWrapper, hMainThread, reinterpret_cast(event)) == 0) { printf("Failed to QueueUserAPC\n"); Pool_Events::free(event); } } //----------------------------------------------------------------------------- DWORD MyGetThreadId() { unsigned long pTID = 0; unsigned long threadId = 0; _asm { mov eax, fs:[0x18] add eax, 36 mov [pTID], eax } ReadProcessMemory(GetCurrentProcess(), (LPCVOID)UlongToPtr(pTID), &threadId, 4, NULL); return threadId; } //----------------------------------------------------------------------------- // Console entry point to save us from having to write a GUI int main(int argc, char * argv[]) { // The physical server that simply churns through data. UDPHighPerformanceServer physicalServer; // The logical server that uses the physical server for high performance // network performance. EdxNetHost logicalServer; UNREFERENCED_PARAMETER(argc); UNREFERENCED_PARAMETER(argv); hMainThread = OpenThread(THREAD_GET_CONTEXT | THREAD_SET_CONTEXT, FALSE, MyGetThreadId()); if(hMainThread == NULL) { printf("Failed to OpenThread: %i\n", GetLastError()); system("pause"); return 0; } // Setup the library, this call will setup the UDP library as well if(InitializeEdxNet() == false) { return 0; } // Create the logic server, allow 1024 incoming connections // and unlimited bandwidth logicalServer.Create(&physicalServer, 1024, 0, 0); // Create our UDP server if(physicalServer.Create(15779, 1, 1, 0, 0) == false) { logicalServer.Destroy(); physicalServer.Destroy(); DeinitializeUDPLibrary(); return 0; } // So we know when to exit hExitEvent = CreateEvent(0, TRUE, FALSE, 0); // We want to set the user data pointer for the // physical server to the logical server so the // two can interact with each other as needed in // our custom packet processing function. physicalServer.SetUserData(&logicalServer); // Register our packet handling callback physicalServer.SetClientToServerCallback(OnClientToServer); // Set a console handler to make our program more flexible and simple. We will be able to // Ctrl + C out of our program rather than have to code in a more complicated GUI. SetConsoleCtrlHandler(ConsoleHandler, TRUE); while(WaitForSingleObjectEx(hExitEvent, 1, TRUE) != WAIT_OBJECT_0) { EdxNetEvent event; int result = logicalServer.Service(event); while(result == 1) { if(event.type == EDXNET_EVENT_TYPE_RECEIVE) { __int64 recvIndex = event.packet->Read(); printf("[%lld] - %i bytes\n", recvIndex, event.packet->GetDataSize()); EdxNetPacket * packet = EdxNetPacket::Allocate(); packet->SetFlags(true, true); packet->Append(recvIndex); // Send the packet to the peer, we do not need to free // the object ourself. event.peer->Send(0, packet); // We will need to deallocate the packet of the event // to keep memory usage as optimized as possible. EdxNetPacket::Deallocate(event.packet); } else if(event.type == EDXNET_EVENT_TYPE_CONNECT) { printf("Client [%X] has connected.\n", event.peer); } else if(event.type == EDXNET_EVENT_TYPE_DISCONNECT) { printf("Client [%X] has disconnected.\n", event.peer); } // Try to see if there is another event waiting for us result = logicalServer.Service(event); } } // Remove the console handler as we no longer need it. SetConsoleCtrlHandler(ConsoleHandler, FALSE); // Cleanup the servers logicalServer.Destroy(); physicalServer.Destroy(); // Finally cleanup the library DeinitializeEdxNet(); // Cleanup resources CloseHandle(hMainThread); // Standard return return 0; } //----------------------------------------------------------------------------- Well, that wraps up this status report. I'm still hard at work and looking forward to being done pretty soon before the month of April is over hopefully! Adieu! ## It is alive! After spending a couple of hours of combining enet into one cpp/h file and then making a few cleanups, I've finally got the basics of Stage 2 of my application in place. It's still far from close to being done, but as of right now the foundation has been laid! I've got a pure enet client talking along with my UDP IOCP enet server! This is great news as I was worried getting to this point would take quite some time. I actually have enet's awesome design to thank for the simplicity of the task. I was able to simply expose some private functions and call them directly and have everything tied together. Here's what the server code looks like (which I am using the Stage 1 library I talked about in the previous entry as the framework) #include "UDPHighPerformanceServer.h" #include "enet.h" //----------------------------------------------------------------------------- // We will use this event as a signal to exit. We simply block the main function // and let the server do all the work in its own threads. This won't be needed // if you have a main loop to execute. HANDLE hExitEvent = 0; //----------------------------------------------------------------------------- // We can use a console handler to detect console closing events to gracefully // handle when the user wishes to exit. BOOL __stdcall ConsoleHandler(DWORD ConsoleEvent) { switch(ConsoleEvent) { case CTRL_LOGOFF_EVENT: case CTRL_C_EVENT: case CTRL_BREAK_EVENT: case CTRL_CLOSE_EVENT: case CTRL_SHUTDOWN_EVENT: { SetEvent(hExitEvent); return TRUE; } } return FALSE; } //----------------------------------------------------------------------------- // Our packet processing function. We have access to everything we need to handle // packets. This function will be thread safe by internal design for each // individual server. void WINAPI OnClientToServer(UDPHighPerformanceServer * server, sockaddr_in & address, UINT16 count, UINT8 * data) { ENetHost * host = reinterpret_cast(server->GetUserData()); ENetEvent event; memset(&event, 0, sizeof(ENetEvent)); host->receivedAddress.host = (enet_uint32) address.sin_addr.s_addr; host->receivedAddress.port = ENET_NET_TO_HOST_16 (address.sin_port); host->receivedData = data; host->receivedDataLength = count; int result = enet_protocol_handle_incoming_commands(host, &event); result = enet_protocol_send_outgoing_commands(host, &event, 1); host->receivedData = 0; host->receivedDataLength = 0; } //----------------------------------------------------------------------------- // Console entry point to save us from having to write a GUI int main(int argc, char * argv[]) { UNREFERENCED_PARAMETER(argc); UNREFERENCED_PARAMETER(argv); ENetHost * server = 0; server = enet_host_create(1024, 0, 0); // The physical server that simply churns through data. UDPHighPerformanceServer physicalServer; physicalServer.SetUserData(server); server->edxlabs = &physicalServer; // Setup the library if(InitializeUDPLibrary() == false) { return 0; } // So we know when to exit hExitEvent = CreateEvent(0, TRUE, FALSE, 0); // Create our UDP server if(physicalServer.Create(15779, 1, 1, 0, 0) == false) { physicalServer.Destroy(); DeinitializeUDPLibrary(); return 0; } // Register our packet handling callback physicalServer.SetClientToServerCallback(OnClientToServer); // Set a console handler to make our program more flexible and simple. We will be able to // Ctrl + C out of our program rather than have to code in a more complicated GUI. SetConsoleCtrlHandler(ConsoleHandler, TRUE); // Block the main thread and let the server do all the work WaitForSingleObject(hExitEvent, INFINITE); // Remove the console handler as we no longer need it. SetConsoleCtrlHandler(ConsoleHandler, FALSE); // Cleanup the server physicalServer.Destroy(); // Finally cleanup the library DeinitializeUDPLibrary(); enet_host_destroy(server); // Standard return return 0; } //----------------------------------------------------------------------------- I am going to have to make enet optimized for this high performance sever as well. It uses linear algorithms for everything right now, but this is definitely a positive sign that I was not expecting so soon. I think I'll call it a night for now and start bright and early tomorrow afternoon! More updates to come! [grin] (I'm too tired to pull off a long post right now, so this is it) 4/12/09 Update 2 Pretty simple day so far. I've been slowly cleaning up enet some and doing some testing. I made a simple latency testing client that sent a 8 byte packet to the server and the server responds with that same data every 100ms. The results of that looked really good. Using reliable packets, I could handle 500 clients pretty fairly across the LAN with response times ranging between (0ms, 300ms]. The values were mostly in the 2 digit range though. I can see the impracticalities of such testing anyways since all those clients are on my laptop and I'm killing the poor thing with all of the work it's doing. Anyways though, I am not disappointed at all with anything so far. I think the performance that enet has will go quite well with my IOCP model, I just need to do a lot more cleaning up and optimizing! The best part is that using enet is just one of the possible layers with the framework, users will be able to use their own layers as well. Gaffner has his own little udp based library in progress as part of his UDP networking tutorial, so as that progresses, I'll see about using those strategies as well. In addition, I've finally got my multithreaded networking concept issues resolved using the same mechanisms that I used them to solve in my library, fancy that. Since the main server loop is running in the main thread and my network event processing function runs in a different thread, I used APCs again to post network events into my server thread and now have the ability to handle everything in only one thread context. The interface between enet and my code is far from being clean or usable though. I'd like to integrate the two more closely, but I don't see that happening as the two code sets are night and day. It will quite literally be making a networking Frankenstein [lol]. That's about it for tonight. The rest of the night will be simply mulling over the cleanup process, I'm not sure how many changes I'll get done but it'll be a drop in the bucket. The original enet project was ~4k total lines of text, I'm down to 2.7k right now. I won't be able to use this enet code for clients either, only my server, but that's 'ok' with me. Oh yea, I still need to reevaluate physics libraries again for my engine. *rubs hands together*, I can't wait until this stuff is done! Two more pictures from the laptop. First is 250 clients and second is 500 clients, executed in 10 client batches: ## UDP High Performance Server: Stage 1 Done! Woo! I've finally finished Stage 1 of the server. There is one oddity though that I can't figure out. The debug version of the library is 700kb. The release version of the library is 1.6MB. WTF! I thought debug libraries should always take up more space than a release. What else is getting compiled into my library in release mode that is not in debug mode?! Both versions work fine as-is and I've not seen any problems, but it's bugging me a little. It's not project configuration settings either since I'm using stock options aside from Multibyte rather than Unicode. Oh well, it doesn't matter it was just unexpected. From here, I will use the library as the base framework for Stage 2, which involves integrating it into enet. I will have to change some of the core architecture of the polling based enet library to work with my event based setup, so I've got fingers crossed it won't be too messy. I'm hoping to get that started tonight, but I'm still trying to figure out where to start. I kind of want to modify enet as little as possible as to not mess anything up there. It's going to be tricky, but I'll manage. I don't have any downloads ready yet though. I might just hold off on posting anything for download until the enet part is done as most people will probably be more interested in Stage 2 rather than Stage 1. I'll base my code project article on Stage 1 if I decide to write it and then come up with a follow up Stage 2 article if time permits and the task is not too hard. I'm excited! With this networking stuff at the half-way point of being done, I'll then be able to take a look back at my engine and figure out a physics direction I want to go with the networking component. I'm thinking I might not stay with Newton after all, but we'll see. Until next time - ## Face Palm - GDWiki Style So I loaded up the GDNet wiki a few minutes ago after making my last journal post. I hit random page a couple times. Lo and behold, look at what I spun: ## GDNet+ Membership Perks Blurb So I took some time this evening to review the negative perks of being a GDNet+ subscriber. This post was just out of pure observation, no real reason I'm posting about this now as compared to never over the past year. Here's what I found: Site no longer exists and it doesn't look like it's been held since 2004? However, I just noticed an event for GDX which just entered its 5th year, which maybe that's the new XGDX? From their site: So I assume the Lab Technology .NET product is no longer being offered. Site no longer exists either. Bulk Ads are still present. 50mb is still there, but no direct FTP. It's been disabled for sometime now. Nice, but a limited system. Couple that with the general performance of the GDNet site and it can be somewhat frustrating at times. These are a bit over-exaggerated. You get 15% off on Course Technology PTR. Everyone gets 20$ off regardless. As for the books from all those wonderful publishers, they are now mostly old books. This means their price has increased greatly as they are no longer being printed. There are new modern books, but you can usually get them a lot cheaper in "like new condition" from Amazon sellers than buying new. Indies are on a budget.

This product might have been alright 10 years ago but today? Your average modern day phone can provide the features this software does. Granted, I've not actually used it, but impressions are everything and I don't get the impression this software would be useful today.

So, yea, the GDNet+ benefits kinda officially stink nowadays. That's not why you sign up for GDNet+ though, you signup to support the best game development forums there are nowadays. Regardless though, these perks are in some desperate need of a Spring cleaning.

I've not noticed anyone else really talk about this since like 04-05 when searching the forums, so I might as well make a post only a few people will actually read as it applies to all us GNet+'ers. Does anyone really care about this or are we all just so used to it that we're indifferent?

I can't wait until the new GD rolls out though, I'm looking forward to lots of improvements to the site. Maybe I should make a journal post suggesting radical new changes just to see what people think. I'm a very opinionated person, I just play the conservative shy guy 99/9% of the time.

## UDP High Performance Server: Stage 1 Preview

I've been talking about it for a little now, and it's almost ready! Stage 1 of the UDP High Performance Server is just about to hit the limelight for critiquing. I cleaned up a few naming conventions today and reorganized the project from one file to 4 files total. I've added a few more comments and double checked for consistency. I've been running massive concentrated UDP tests from my laptops and the server still barely breaks a sweat (of course, I don't have any real latency tests implemented yet, but the server doesn't croak and die on me)

Project Goals:

1. High performance industrial grade scalable networking solution for modern Microsoft Windows platforms.

2. Simple yet flexible non over-engineered architecture that didn't impact performance and did not impose any restrictions on the end user.

3. Easy to use API and trivial integration into existing projects.

4. Open source project for the networking community to learn the arcane nature of properly using IOCP alongside UDP while being able to see well documented code and not something that simply worked.

5. A framework to base my future networking endeavors upon.

At this point, I believe I have satisfied 1,2,3 and partially satisfied 4 and 5. I've not yet released the code yet, so 4 is still pending, and I still need to come up with a TCP based solution as well for #5. Having done UDP first though has taught me a lot and it should be easier going back and making a better TCP version than I had before.

Since it's really (early) now and I'm dead tired, I'll post the public API and a simple example program showing a preview.

UDPHighPerformanceServer.h

#pragma once

#ifndef UDP_HIGH_PERFORMANCE_SERVER_H_
#define UDP_HIGH_PERFORMANCE_SERVER_H_

//-----------------------------------------------------------------------------

#include
#include
#include

//-----------------------------------------------------------------------------

// Forward declarations
class UDPHighPerformanceServerData;
class UDPHighPerformanceServer;

//-----------------------------------------------------------------------------

// This is a function pointer typedef that we will use to register a function
// to receive packets from clients sent to the server with in our main program.
typedef void (WINAPI * OnClientToServerFunc)(UDPHighPerformanceServer * server,

//-----------------------------------------------------------------------------

class UDPHighPerformanceServer
{
private:
// Our PIMPL pattern in action
UDPHighPerformanceServerData * internalData;

// Explicitly disable assignment
UDPHighPerformanceServer & operator = (UDPHighPerformanceServer & rhs);

// Explicitly disable copy construction
UDPHighPerformanceServer(const UDPHighPerformanceServer & rhs);

// Allow the user to assign a user data field to this object
void * userData;

public:
UDPHighPerformanceServer();
~UDPHighPerformanceServer();

// Create the UDP server. Returns true on success and false on failure.
// numberOfConcurrentThreads - How many active IOCP threads are allowed (0 - auto detect)
// numberOfWorkerThreads - How many worker threads to create (0 - auto detect)
// maxPendingRecvs - How many pending receives to post (0 - auto detect, 2500 - 32bit max, 32767 - 64bit max)
// refillPendingRecvs - How many pending receives can be left before refilling the queue (0 - Use 1/2 of maxPendingRecvs)
bool Create(UINT16 port, UINT32 numberOfConcurrentThreads = 0,
UINT32 numberOfWorkerThreads = 0, LONG maxPendingRecvs = 0, LONG refillPendingRecvs = 0);

// Close down the server and free internally used resources.
void Destroy();

// Sets our callback function. If we wish to not have one, we may
// pass 0 to use a default internal function.
void SetClientToServerCallback(OnClientToServerFunc function);

// Send to a specific ip/port. Our function will cache the addresses
// so multiple calls will not incur the same overhead that a first
// call for a specific address might have.
bool SendTo(const char * ip, USHORT port, UINT16 count, UINT8 * data);

// Send to a specific address. We will use this most of the time as we
// are given a sockaddr_in object in our packet processing function.

// Gets/Sets user data for a custom context field for the object
void * GetUserData() const;
void SetUserData(void * data);
};

//-----------------------------------------------------------------------------

// Returns true on success and false on failure. You only should call this
// function once before using the library.
bool InitializeUDPLibrary();

// Cleans up the library. You should only call this function once at program
// exit.
void DeinitializeUDPLibrary();

//-----------------------------------------------------------------------------

#endif

Server.cpp

#include "UDPHighPerformanceServer.h"
#include

//-----------------------------------------------------------------------------

// We define a logical server class to encapsulate all of our networking
// endeavors and keep an OO-styled approach to the system. This is not required,
// but this demo shows that it is still possible.
class LogicalServer
{
private:

public:
LogicalServer()
{
}

~LogicalServer()
{
}

void OnClientToServer(UDPHighPerformanceServer * server,
{
printf("[%X] %i.%i.%i.%i:%i\n", PtrToUlong(server),
for(WORD x = 0; x {
printf("%.2X ", data[x]);
if((x + 1) % 16 == 0)
printf("\n");
}
printf("\n");
}
};

//-----------------------------------------------------------------------------

// We will use this event as a signal to exit. We simply block the main function
// and let the server do all the work in its own threads. This won't be needed
// if you have a main loop to execute.
HANDLE hExitEvent = 0;

//-----------------------------------------------------------------------------

// We can use a console handler to detect console closing events to gracefully
// handle when the user wishes to exit.
BOOL __stdcall ConsoleHandler(DWORD ConsoleEvent)
{
switch(ConsoleEvent)
{
case CTRL_LOGOFF_EVENT:
case CTRL_C_EVENT:
case CTRL_BREAK_EVENT:
case CTRL_CLOSE_EVENT:
case CTRL_SHUTDOWN_EVENT:
{
SetEvent(hExitEvent);
return TRUE;
}
}
return FALSE;
}

//-----------------------------------------------------------------------------

// Our packet processing function. We have access to everything we need to handle
// packets. This function will be thread safe by internal design for each
// individual server.
void WINAPI OnClientToServer(UDPHighPerformanceServer * server, sockaddr_in & address, UINT16 count, UINT8 * data)
{
LogicalServer * customServer = reinterpret_cast
(server->GetUserData());
}

//-----------------------------------------------------------------------------

// Console entry point to save us from having to write a GUI
int main(int argc, char * argv[])
{
UNREFERENCED_PARAMETER(argc);
UNREFERENCED_PARAMETER(argv);

// Our logical server that uses the physical server's data
LogicalServer logicalServer;

// The physical server that simply churns through data.
UDPHighPerformanceServer physicalServer;
physicalServer.SetUserData(&logicalServer);

// Setup the library
if(InitializeUDPLibrary() == false)
{
return 0;
}

// So we know when to exit
hExitEvent = CreateEvent(0, TRUE, FALSE, 0);

// Create our UDP server
if(physicalServer.Create(15779) == false)
{
physicalServer.Destroy();
DeinitializeUDPLibrary();
return 0;
}

// Register our packet handling callback
physicalServer.SetClientToServerCallback(OnClientToServer);

// Set a console handler to make our program more flexible and simple. We will be able to
// Ctrl + C out of our program rather than have to code in a more complicated GUI.
SetConsoleCtrlHandler(ConsoleHandler, TRUE);

// Block the main thread and let the server do all the work
WaitForSingleObject(hExitEvent, INFINITE);

// Remove the console handler as we no longer need it.
SetConsoleCtrlHandler(ConsoleHandler, FALSE);

// Cleanup the server
physicalServer.Destroy();

// Finally cleanup the library
DeinitializeUDPLibrary();

// Standard return
return 0;
}

//-----------------------------------------------------------------------------

That's what I have right now code wise to create a IOCP powered UDP server that scales based on your hardware and efficiently manages all network events. For users that need a single threaded design, they will have to implement some synchronization between the logical server and main thread and any additional threads they might create (such as a timer thread). I don't quite like the limitations of a "polling" system to get packets, so this design allows users to implement that using simple synchronization.

Well, that's it for this short preview. I will probably post again either this afternoon or hold out until this weekend.

## More progress being made with network code!

It's been three days since my last post on the networking stuff, time for more updates!

I've spent the past three days making the framework solid. By solid I mean reaching every detail on MSDN for the API functions I am using to make sure I am doing things the correct way. It's been quite a bit of work, but I'm in love with my code now; it's sexy!

I did find a handful of bugs today in regards to how I was using various APIs. In some cases, I was using variables on the stack that needed to be more persistent, mishandling specific network events, and had a few minor cleanup bugs. All of that has been fixed now and I am confident I am correctly using the APIs correctly.

For this framework, I am using the Boost Singleton Pool to handle memory. The reason for this is because of the high frequency of allocations and deallocations in my code, I need a pool. I was reading some of the caveats to using the library, but I find it overall acceptable to use and include in my framework.

I had a lot of "TODO: Log error" comments in my code and I've eliminated all those today. I just went with a simple printf styled log function that gets the job done. I did not want to come up with a heavy weight solution simply because that will be up to anyone who uses the code to handle errors their own way, so I did want something in place that 'works'.

Up until yesterday, I could only receive on my server. I've now added the ability to send as well and it looks like it's working so far. I had a little trouble with the WSASendTo function as completion routines that I'm trying to figure out. I posted on the MSDN network mailing list and hoping for an explanation. You can read that thread here: Completion routine causes WSASendTo to fail when using UDP w/ IOCP. I don't need it, but it's a thorn in the side until I can figure out why it fails.

I think one of the best aspects to this framework are my comments. In the past I have been notorious for using oversimplified comments that were pointless and just wasting space. You know, the ones like: "// Add x to y", "// Close the handle", etc... Now, my comments have purpose and meaningfully contribute to the code. Rather than explain the code, they explain the logic, which is what comments are supposed to do. In addition, I've added some links where needed to backup design choices. Overall, I feel people will be able to actually learn from this code rather than have something that just works.

Stage one of the framework is almost done. This initial version I am working on is to serve as a framework shell for high performance, scalable UDP. All it will be able to do is effectively process and send packets. There will be no custom protocol over UDP implemented. The reason for this is to give people something that works well that they can add their own custom logic to and expand it from there.

Once stage one is completed, it will serve as the foundation for stage 2, which will be implementing the enet protocol into the server. I'm going with enet because it is pretty simple, compared to something like TNL, has a very nice license, and it works. Simple enough, right? I do want to make some changes to the protocol as well as how the code is setup. I've spent some time in the past couple of days messing with that library as well, tracing code and putting the pieces together. There are a few things in there that make me go UGH, but overall it's not that bad.

I don't know how long that task will take me now. At first I thought I could get it done in a week or so, but I'm thinking it's going to take a little longer understand the entire library and reimplementing it how I want to. At least I'll have something to keep me busy though! From here, I plan on reviewing my framework some more looking for any possible problems or lack of comments. I've made sure things are consistent and mostly "modern" rather than hackish C as I've done with things in the past.

I'd love to write a CodeProject article on this stuff as the current articles are really lacking and have a lot of flaws in there, but I don't think I have the time for that. I'll have to see though. I'm just looking over the code for the framework right now, and it's amazing how little code is required for something so powerful! The entire file right now is just under 1000 lines. However, that includes a massive amount of comments and proper spacing. I'd not be surprised if the bare framework took up less than 300 lines of code if even that!

That wraps up this status update. I will post some more updates on progress maybe at the end of this weekend. I still need to get back to my other IOCP TCP thread and fix that code up too, but that can wait, I love this UDP too much to stop.

Cheers! [smile]

## 4/5/09 Status Report: The hard-hat is still donned

Since it has been a little while since I've had a status update I think I'll post one now.

There has not been a Game Engine Entry #2 posted yet because I am still working on this networking stuff. The reason I switched gears so suddenly is because I want a game engine that is has a good solid network architecture to drive it, since graphics are not going to be the "selling point" so to speak. After seeing my limitations with my older network code that used TCP and asynchronous sockets via Win32 HWNDs, I decided to take the IOCP plunge. I literally spent weeks doing research and development with various examples before writing my own.

When I did write my own though, boy was I in for a surprise! I knew TCP was somewhat limited to UDP, but I really discovered the limitations when I went through the testing cycle. Given those issues with really needing to run on a server and have way more clients at my disposal to test with, I was about to buckle down and make some investments in more hardware. However, Ysaneya posted a reply in my thread asking if the problems I was having were still persistent with UDP.

I really have to give it to him for bringing up that observation. Being the overly motivated person I am, I was very inclined to see, so I started porting my code over to UDP where I had some things to learn along the way. Initial testing showed amazing results that really looked too good to be true. No way I should be able to get that type of performance from UDP, or so I was thinking. I took a few days and totally rewrote my UDP code with IOCP to make it more realistic and efficient, as the last version was built on top of TCP code.

I ran my tests again and sure enough, the original performance I was seeing was "true" and "accurate". It is simply amazing the differences between the two protocols running on localhost. With TCP, I over utilized the hardware/software I have so the server doesn't work right. With UDP, I ran out of resources and the server still chugged along fine. Granted I have 32-bit/64-bit limitations I have to work around now, I was very pleased with the output and decided I will be investing in a UDP framework rather than TCP.

I ran a simple test of 2,000 UDP clients overnight for 8 hours and it went great. Only about 300 dropped packets (or packets that never made it to the server) for each connection was the average. Service time delay between the connections was just a little under 20ms. Likewise, I ran a ~20,000 UDP client test just to see how that'd go and the limitations of 32bit windows was apparent. I could only handle 2500 events at a time which was a big bottleneck for the server. In addition my 64-bit laptop was hold 17k of those connections and the 32-bit xp laptop had bout 3k, so it was very biased.

Anyways, I've spent a good days worth of research this weekend looking at how I want to pick up a few more 64-bit vista desktop to run tests on. I'm really killing my laptops, so I want to replace those testbeds. I really wanted to build a small army of them, but it's simply too many parts to want to deal with any possible DOAs or the like. I can build a sweet slim microatx desktop that would suit my testing needs for between \$300 (bare minimal) - 381(fully loaded). However, I see too much risk in ordering multiples of all that hardware, even though I'm confident in NewEgg. As a result, I think I'll just pay the extra "convenience cost" for a premade desktop from Newegg and order multiple of those to cut down on risk and time spent building.

Now that my simple UDP IOCP code is underway, I find it to be time to look towards the future. I need a UDP library that is useful for games for the engine. I've decided I will take enet and integrate it into my code to make it a Win32 UDP library on steroids! I've just now started as of this post on the process of learning the internals of enet. I'm going through code and GameDev threads that contain the keyword. I hope sometime by the end of this week I have something "working" that does what it's supposed to.

I'm really looking forward to this project. I used to be complacent with using TCP and how much work it cuts down for you, but I'm really excited about using UDP now.

That's about it for this status update, hope you enjoyed [smile]

## Networking Blurbs #1

As a few people might have noticed, I've been working on Trying to get my IOCP code on par with expected performance. I'm coming across a few little things here and there that I want to have in one spot before I post and what better places than my Journal!

I'll be adding stuff here and then condense it into a post later in the upcoming week.

1. Maximal number of overlapped receive requests on a socket:
Windows XP 32bit = 2,500
Windows Vista Home 64bit = 32,767

I was writing a simple system to track the total overlapped events on my main socket when I wondered how many I could actually have. I added in the logic to post as many overlapped WSARecv(From) events on my socket and those were the results. I'm willing to bet it's a 32bit and 64bit limitation. I'll post a code sample later on of showing how to calculate it yourself if you get stumped. Here's the method:
1. Setup your regular socket for a server
2. Loop from 1 - INFINITE and post overlapped WSARecv(From) requests on the socket
3. WSARecv(From) will return -1 each time. You should be getting an error code of ERROR_IO_PENDING from WSAGetLastError. You know that event was successfully queued. As soon as you get an error of WSAENOBUFS though, then you have hit the limit and know how many you can support based on the loop counter.

Pretty simple!

2. Coming soon!

## Game Engine Entry #1

For the time being I will be working on my little game engine project. I'll periodically add updates as to what I am doing and how things are going here just to get some thoughts out. I really want to get my own site and blog setup, but I've failed at committing to one so many times I won't even bother again for now. I'll just jot stuff down here!

I won't get too much into what I hope to accomplish this this project. Right now, it really is just a time passer to keep active and expand my horizons a little. I do not plan on writing everything from scratch either. I plan on taking as much middleware as possible and plugging it in to form the engine. On that note, I should start out with the resource I am currently using.

Framework - SFML
Terrain - OpenGL 2.0 Shading Language Terrain Texturing Demo.
Camera - Simple gluLookAt camera
Viewport - Multiple Viewports
Physics - Newton
Secondary Input - Xinput
Math - Configurable Math Library
Model Format - Wavefront Obj via GLM

So far, the most time has been spent understanding the camera. I was mixing concepts that I should not have been and as a result had written 3 different camera classes that I scrapped. The camera system I have now is ok for testing, but needs big improvements. The terrain and physics both were a little time consuming as well. Porting the terrain system into my project was pretty trivial, but getting it to work with Newton was a hair pulling experience. It only took a few hours though. The viewport and XInput were cake.

As of right now, I have a height mapped terrain (texturing done on the gpu) that has a heightmap collision feature from newton. I can create newton objects and they interact correctly with the terrain. I can move around the terrain using my camera system as well. The camera system is not physics based right now, it is only height mapped based. The code is extremely messy, but most of that is due to how I literally just got physics added as i wrote this post. I will have to spend some time understanding Newton and wrapping up some of its goodies for easier use.

Looking through the Newton code, I'm going to have to spend a lot of time cleaning up the project. The coded uses Glut and is done in Visual Studio 6, ugh. It also uses its own set of math libraries and stuff, so I'll have to rip out all of that and add in CML. Then, I have to factor in how it sets up its own rendering logic and either use that myself or create my own. At least it works though. Maybe I will put off cleanup until later and just work on adding more features first.

Here's a simple video. I know it doesn't look like much and it's really not, consider how much code of other people I am using [lol] Anyways, it's progress though and a decent start. I included all the links above to resources I am using, so right now anyone can implement exactly what I have.

Zipped Video (14 MB)

Where does that leave me now. I'd like to add in some simple networking, but I don't have enough done to utilize it correctly at this point. I'll wait for that until later. I need to cleanup all of Newton but that can wait too since it 'works' good enough as-is. Audio I can do via SFML, so I'm ok in regards to that. I don't need any fancy graphics stuff either, so I don't have to worry about shadows and lightning at this point.

Darn, all that really leaves right now is the clean up and then expanding the API. I don't think I can do anything else without getting things setup how I need them to. That means most of my work over the next week is learning Newton and fixing that up. I really should have all this stuff planned out, maybe I'll get some plans together after the cleanup.

That wraps up this first status entry, over and out.

## Happy March!

Wow, 2009 is already 16% done. Time goes by so fast nowadays it seems. Since it is a start of a new month, I figured it would be a good time to lay out some objectives for the rest of the year and ramble a bit.

The most important issue for me right now is direction. What should I invest my time and energy into? What do I need to focus on right now at this relatively early stage of my career? Should I be working for the moment, the next couple years, or for the long run?

Starting out, I figure I really need to expand on my C++ knowledge. Technically speaking, I don't know that much; I never just studied it. I always just did it. As a result, my C++ work is a mix of C/C++ concepts that works, but is not always the "right" way of doing things. This issue is not hard to solve, I already recognize my deficiency in modern C++ knowledge and have invested in a number of books to learn what I need to know. I've got more than enough right now to help updated my C++ knowledge, but I plan on getting a few more resources. This issue is not really a problem, just absorbing it all and then applying it to my practices. Easy enough for me.

I also need to learn more languages to be able to efficiently expand my C++ projects. I already decided I should be learning more Lua to use alongside the projects. I picked up two main Lua references to help me along the way. However, I also need a second language to use independently of C++. A lot of recommendations go towards Python. Perhaps I'll go that route. On the other hand, I probably should get really proficient in C# and the .Net platform. As I am a Windows user and will be for the unforeseeable future, I need to have those skills. I probably will continue to put off Python.

I love games. I'd really like to make my own games, but I don't know if I have the drive to do it successfully. The reason is, I don't have anyone in my life that I share that magical bond with to be able to take on the world. You know how a lot of people have a best friend they do everything with, or someone they talk to that shares their same interests and they can work with? Well I've never had that and at my age with the type of personality I have, it seems impossible.

As a result, I am reduced to working alone. The immediate drawback to that is a lot of projects out of my immediate reach due to how efficient I would have to work to be able to have a chance to implement them. I've recognized that over the past few years, so I did take lots of time learning how to work smarter and not harder. However, there is still only so much one person can do. I've also noticed I don't work so well trying to implement my own ideas. All of this combined, makes my future in game development really iffy. I want to do it, but I can't myself at this point.

What else is there then? Well, I could always go the route of writing game specific mods, utilities, and bots. I spent 06-08 in that area and landed in a very fortunate position from 07-08. I learned a lot and had tons of fun. Why aren't I still doing it then? Well, the opportunity just ran out.

Things don't last forever so it was time to move on and rather than waste time doing nothing, I switched gears back into Gamedev related topics and picked up some more knowledge from Jan-Feb in regards to game engines. That's me, always keeping busy, always trying to stay ahead of the curve and never settle for anything less than the best.

The problem with this route though is it is very volatile and like anything, has a very low rate of success. In addition, there aren't many games out or are coming out for at least a year that I could get into as far as I've seen. I could go back to the game where I learned it all from, but I don't know if I'm comfortable with that. Sometimes it's good to move on and just never look back.

I'd really like to write a book, but I don't have enough material to cover in it at this point. I sure feel like I've given out enough advice and information to be able to write one though. It would just be a mix on my personal philosophy and some programming stuff. The book idea aside, I'd be in a better position to write tutorials and articles for specific technologies. That kind of stuff I doubt I could get paid for, but I would enjoy it, for a while at least.

That really leaves me with one last thing on my mind of directions to travel, and that is to write my own game engine. It would definitely not be a cryengine2 or unreal3 killer or anything of that nature, but it would be designed around providing a simple api and providing functionality rather than end results. What I mean is, it seems the rendering of an engine is one of the most important features of indie to small business oriented engines nowadays (
To me, it seems the engines try to make everything look great, but as a consequence, they suffer from a lot of other shortcomings with the functionality of the engine and the programming interface. I know a perfect engine is not possible at such a low price tier range, but I would focus on everything but the graphics rendering, which is the only part of an engine I am not apt enough to handle myself.

I already started working on the basics of one using SFML and OpenGL. I spent a few days working through camera examples trying to get the one that I liked and eventually got one, but I will have to rewrite it anyways. I'd like to go the engine route, but I have to look at it long term, what am I going to be getting in return from all the time invested? That is a big uncertainty and while I'd love to write my own simple mmo-engine to get rich off of, it's still quite an iffy direction to go.

Ultimately, the big question is how am I going to make money and "be happy" without a job. That question is incredibly hard for me to answer because I am not a money motivated individual. I'd rather code for free and share knowledge and resources the rest of my life, if not for the fact I'd not survive for long on that path. Doing that stuff makes me happy, helping people makes me happy, so it's hard for me to put a price on things that I want to just do for people. I don't feel it's hopeless though, no way.

Well, that about wraps up things for now. I think I just need to brainstorm out all the possible things to do, spend a few days thinking them over for good measure and then pick one and go full steam on it. It's not easy though, for reasons that are hard to convey, but hard work and dedication without giving up will eventually get me where I want to be, even if I don't know where that is right now [smile]

## Viva la revolucion!

I've finally committed to a game engine. I decided on the Leadwerks Engine. There were many numerous reasons why I decided to give this engine a try to develop my 1st project in:

* Video / PDF tutorials - Quite a lot of tutorials that show basics of using the engine. What was impressive was the strange connection I had to seeing the code, I didn't feel alienated by the setup of things. I wasn't left thinking, "I wonder how the programming goes using this engine" as I was with many others.

* Site Layout - The site was very user friendly and looks clean. I could find most of everything I wanted to know there. Very simple to navigate and plenty of resources on there.

* Forum Support - The lead developer is active on the forums. Having helped ran sites before, I know the demands of handling forums + business + development, so it's a good sign of how serious the project is as the developers are to committing to it.

* Evaluation Kit - This is nothing new of engines, but the matter of the fact is, you kind of associate these with "serious" engines. So when you come across an engine that is not "widely" talked about, like Torque, it's easier to place confidence in it.

I clicked through the top 10 commercial engines on DevMaster's engine list and took a look through all of the projects sites and this projects was the one that stood out the most. I saw a few engines that looked really nice and all, but none of them made me feel like "wow, I actually want to use this product". Even if what I want to so fails or things turn out not so good, it wouldn't matter.

I was getting ready to shell out for Torque but wasn't quite sure confidence wise of trying it. It looked like it had a lot of good stuff in it, but I was still on the borderline of that decision. After looking over the Leadwerks site and forums for a few hours, I was solidly sold. I want to use this product and I want to finally make a game with it.

Before I started all my engine research today, I took another look at a possible DIY approach. I've always liked working on my own frameworks and what have yous, but as a result, I've never had a working game. I've only learned about tool design in practice and nothing about actual game designing. Now that I've spent all of 2008 understanding network interactions in TCP based MMORPGs, I'm eager to prototype out ideas, but I simply don't have the means to.

That's when I decided, I need to stop the impracticalities of trying to have complete control over code and actually get something done using existing middleware. I mean without a doubt, I have the time, patience, and discipline to actually do a game form start to finish, but I never focused on that, ever.

I just wanted to make tools or frameworks that can be used to make games easier and while those are good learning experiences, learning experiences alone aren't going to pay bills. Trying to show off YAUF (yet another useless framework), having a working game framework to prototype out ideas and see them in action is what I need now.

What I've learned in the past year is that, when I write practical programs that have a specific use, I end up learning more and having concepts I reuse in ways greater than any framework or 'generic' code I've ever written before. In fact, I don't think I even use any of my old generic framework code since it was focused on SDL or wrapped a few OpenGL things (never got into DX).

All of the work I've done in the past year though, will be applicable and useful for 'a while'. I've built up a lot of experience in the game security domain and am looking forward to apply those principles to my own programs. I should just look for some security consulting for games, but those opportunities are hard to come by. I'm not an expert in all the theory, but I have become quite efficient in breaking through protocols [grin]

Anyways, I should post some game design ideas later of what I plan on making once I get the engine. I want to get this project off the ground fairly fast, I want to out do what I've done in '08, which I'm not sure I'll be able to hit that peak again in my life, but I still want to try.

Over and out -

## edxTCPNetwork - Almost Done! !

I've been hard at work the past few days improving and bug testing my TCP network library. Right now it's going great! I've finally gotten it so it is nicely organized and does not make use of C-style callbacks. Everything is done through an OOP design, so it is very easy to use to add networking capabilities to a project. Stay tuned for a post in the networking forum soon asking for a critique on it! Here's a preview of a simple client and server using the library.

Server

#ifdef _DEBUG
#include "../debug/edxTCPNetwork.h"
#pragma comment(lib, "../debug/edxTCPNetwork_d.lib")
#else
#include "../release/edxTCPNetwork.h"
#pragma comment(lib, "../release/edxTCPNetwork.lib")
#endif

#include
#include

//-----------------------------------------------------------------------------

// This class represents the simplest server class you may have.
class SkeletonTCPServer : public edx::TCPServer
{
public:
// Default ctor
SkeletonTCPServer()
{
}

// Default dtor
~SkeletonTCPServer()
{
}

// User function called when a client connects to the server
bool TCPOnConnect(edx::TCPClientInterface * client, const CHAR * ip, WORD port)
{
printf("[SkeletonTCPServer]Client [%X] connecting from %s:%i\n", PtrToUlong(client), ip, port);
return true;
}

// User function called when a client sends data to the server. Should return -1
// if the stream of data is not processable (needs more data) or a value > 0
// that represents how much data was processed.
int TCPOnProcessStream(edx::TCPClientInterface * client, WORD size, LPBYTE stream)
{
printf("[SkeletonTCPServer]Received %i bytes from client [%X].\n", size, PtrToUlong(client));
return -1;
}

// User function called when a client closes the connection
void TCPOnClose(edx::TCPClientInterface * client)
{
printf("[SkeletonTCPServer]Client [%X] disconnected.\n", PtrToUlong(client));
}

// User function called when a client encounters an error on the server
// (not required, but a good idea)
void TCPOnError(edx::TCPClientInterface * client, CONST CHAR * file, CONST CHAR * function, CONST INT line, CONST TCHAR * error)
{
printf("[SkeletonTCPServer]Error generated on client [%X] - %s\n", PtrToUlong(client), error);
}
};

//-----------------------------------------------------------------------------

// Entry point
int main(int argc, char * argv[])
{
UNREFERENCED_PARAMETER(argc);
UNREFERENCED_PARAMETER(argv);

// Server object
SkeletonTCPServer server;

// Setup Winsock
if(edx::InitializeWinsock(2,2) == FALSE)
return 0;

// Try to create the server on port 16000
if(server.Create(16000) == false)
{
// Clean up the server
server.Destroy();

// Cleanup Winsock
edx::DeinitializeWinsock();

// Standard return
return 0;
}

// Just let the server run for 15 seconds so we can start a client
Sleep(15000);

// Clean up the server
server.Destroy();

// Cleanup Winsock
edx::DeinitializeWinsock();

// Keep output on screen
system("pause");

// Standard return
return 0;
}

//-----------------------------------------------------------------------------

Client

#ifdef _DEBUG
#include "../debug/edxTCPNetwork.h"
#pragma comment(lib, "../debug/edxTCPNetwork_d.lib")
#else
#include "../release/edxTCPNetwork.h"
#pragma comment(lib, "../release/edxTCPNetwork.lib")
#endif

//-----------------------------------------------------------------------------

// This class represents the simplest client class you may have.
class SkeletonTCPClient : public edx::TCPClient
{
public:
// Default ctor
SkeletonTCPClient()
{
}

// Default dtor
~SkeletonTCPClient()
{
}

// User function called when a client connects to the server
bool TCPOnConnect(edx::TCPClientInterface * client, const CHAR * ip, WORD port)
{
printf("[SkeletonTCPClient]Client [%X] connecting to %s:%i\n", PtrToUlong(client), ip, port);
return true;
}

// User function called when a client sends data to the server. Should return -1
// if the stream of data is not processable (needs more data) or a value > 0
// that represents how much data was processed.
int TCPOnProcessStream(edx::TCPClientInterface * client, WORD size, LPBYTE stream)
{
printf("[SkeletonTCPClient]Received %i bytes from server [%X].\n", size, PtrToUlong(client));
return -1;
}

// User function called when a client closes the connection
void TCPOnClose(edx::TCPClientInterface * client)
{
printf("[SkeletonTCPClient]Client [%X] disconnected.\n", PtrToUlong(client));
}

// User function called when a client encounters an error on the server
// (not required, but a good idea)
void TCPOnError(edx::TCPClientInterface * client, CONST CHAR * file, CONST CHAR * function, CONST INT line, CONST TCHAR * error)
{
printf("[SkeletonTCPClient]Error generated on client [%X] - %s\n", PtrToUlong(client), error);
}
};

//-----------------------------------------------------------------------------

// Entry point
int main(int argc, char * argv[])
{
UNREFERENCED_PARAMETER(argc);
UNREFERENCED_PARAMETER(argv);

// Client object
SkeletonTCPClient client;

// Setup Winsock
if(edx::InitializeWinsock(2,2) == FALSE)
return 0;

// Try to connect to our local server
if(client.Connect("127.0.0.1", 16000, false) == false)
{
// Clean up the client
client.Destroy();

// Cleanup Winsock
edx::DeinitializeWinsock();

// Standard return
return 0;
}

// Keep it connected for 5 seconds
Sleep(5000);

// Clean up the client
client.Destroy();

// Cleanup Winsock
edx::DeinitializeWinsock();

// Keep output on screen
system("pause");

// Standard return
return 0;
}

//-----------------------------------------------------------------------------

Pretty simple if I might say! What is not shown is any "packet" handling because there is no integrated handling to the library. The approach I took was to make a library that would do all of the hard work of using TCP, then expose a function that gives you data as it's received and then you can implement your own custom data handling from there without being tied to the network code specifically.

Right now I am working on my generic packet reader and packet builder classes and then a simple protocol to sit on top of the stream processing to show why I am proud of the design. Stay tuned!

## Network Code Updated!

The more I time I spend on this project the more subtle bugs and flaws I come across in my code [lol]. Today I fixed a few issues. The first was with the Connect function in the client. Originally, I had it so it returned immediately due to the async nature of it, and there was really not way of knowing if it actually connected or not! To fix this I implemented the proper error handling code in the FD_CONNECT event. I added a member variable iConnected to act as a connection flag. When the client starts out it is -1 to mark as, waiting to connect. If the connect fails, it will be set to 0, otherwise if the connect succeeded it will be 1. In the Connect function, I added a loop at the end to wait for the flag to change from -1. A simple, while(iConnected == -1) Sleep(100);. This will block for a few seconds, depending on how long the connection takes, but it is the most intuitive approach, since you obviously want to know if the connection was made before continuing on in your program!

The next thing I fixed was making the project mutlibyte and unicode compatible. I.e. the default project settings that revert the project to Unicode, the code will compile with. Normally, I just change the options to multibyte and design for just that setting. I made use of the TCHAR family of functions, which is windows specific, but my library is Win32 specific, so all is well. That is not to say the library is 100% unicode compatible, but it compiles under the settings.

I also got rid of all warnings on C4 settings of debug and release. This matters to some people, so I've made sure to get all the little things taken care of. I made a few general cleanups along the way, mostly removing some older code no longer needed ever since I removed the accept thread and implemented that asynchronously. The only warnings left are the ones caused by not declaring _CRT_SECURE_NO_WARNINGS, so I kind of cheated with that, but I implemented error reporting, so it's a fair trade off. I want the code to work on VS 2003 too, so that's why I didn't update to _s functions.

Now, I think I'm to the point where the core library is 'done' for TCP. I need to implement a UDP version later on, but for now I am just doing TCP. Here's a bare minimal server example of using the code:

#ifndef EDX_NETWORK_H_
#include "../edxNetwork.h"
#endif

#ifndef _INC_CONIO
#include
#endif

#ifndef _INC_TCHAR
#include
#endif

//-----------------------------------------------------------------------------

class cServer : public edxServer
{
private:

public:
// Default ctor
cServer()
{

}

// Default dtor
~cServer()
{

}

// Function called when a client has connected to the server
void OnConnect(SOCKET s)
{
UNREFERENCED_PARAMETER(s);
_tprintf(TEXT("cServer::OnConnect\n"));
}

{
UNREFERENCED_PARAMETER(s);
}

// Function called when a socket is ready for writing
void OnWrite(SOCKET s)
{
UNREFERENCED_PARAMETER(s);
_tprintf(TEXT("cServer::OnWrite\n"));
}

// Function called when a socket was closed
void OnClose(SOCKET s)
{
UNREFERENCED_PARAMETER(s);
_tprintf(TEXT("cServer::OnClose\n"));
}

// Function called when an error occurs
void OnErrorFunction(CONST CHAR * file, CONST CHAR * function, CONST INT line, CONST TCHAR * error)
{
UNREFERENCED_PARAMETER(file);
UNREFERENCED_PARAMETER(function);
UNREFERENCED_PARAMETER(line);
UNREFERENCED_PARAMETER(error);
}
};

//-----------------------------------------------------------------------------

// Entry point
int main(int argc, char * argv[])
{
UNREFERENCED_PARAMETER(argc);
UNREFERENCED_PARAMETER(argv);

// Setup Winsock
if(edxInitializeWinsock(2,2) == FALSE)
return 0;

// Allocate memory for a new server object
cServer * server = new cServer;

// Create the server on port 16000
server->Create(16000);

// Do nothing while the server loops
while(!_kbhit())
{
Sleep(500);
}

// Cleanup the server
server->Destroy();

// Free memory
delete server;

// Cleanup Winsock
edxDeinitializeWinsock();

// Standard return
return 0;
}

//-----------------------------------------------------------------------------

As you can see, it is at the lowest layer. You would from here have to implement your own data stream processing how you needed it. The logic is pretty simple, so I'll write a simple wrapper class that shows an example, like the cProtoServer in the previous entry. You may also note that no clients are tracked and there is no 'client data' stored, all those things are for the derived class to implement, so the core code just provides the means to get the data easily without messing with the Win32 layer.

A client is equally as bare:

#ifndef EDX_NETWORK_H_
#include "../edxNetwork.h"
#endif

#ifndef _INC_CONIO
#include
#endif

#ifndef _INC_STDIO
#include
#endif

//-----------------------------------------------------------------------------

class cClient : public edxClient
{
private:

public:
// Default ctor
cClient()
{

}

// Default dtor
~cClient()
{

}

// Function called when a client has connected to the server
void OnConnect()
{
_tprintf(TEXT("cClient::OnConnect\n"));
}

{
}

// Function called when a socket is ready for writing
void OnWrite()
{
_tprintf(TEXT("cClient::OnWrite\n"));
}

// Function called when a socket was closed
void OnClose()
{
_tprintf(TEXT("cClient::OnClose\n"));
}

// Function called when an error occurs
void OnErrorFunction(CONST CHAR * file, CONST CHAR * function, CONST INT line, CONST TCHAR * error)
{
UNREFERENCED_PARAMETER(file);
UNREFERENCED_PARAMETER(function);
UNREFERENCED_PARAMETER(line);
UNREFERENCED_PARAMETER(error);
}
};

//-----------------------------------------------------------------------------

// Entry point
int main(int argc, char * argv[])
{
UNREFERENCED_PARAMETER(argc);
UNREFERENCED_PARAMETER(argv);

// Setup Winsock
if(edxInitializeWinsock(2,2) == FALSE)
return 0;

// Allocate memory for a new server object
cClient * client = new cClient;

// Connect to the server
if(client->Connect("127.0.0.1", 16000) == false)
{
_tprintf(TEXT("Could not connect to the server!\n"));
}
else
{
// Do nothing while the server loops
while(!_kbhit() && client->IsConnected())
{
Sleep(500);
}
}

// Cleanup the client
client->Destroy();

// Free memory
delete client;

// Cleanup Winsock
edxDeinitializeWinsock();

// Standard return
return 0;
}

//-----------------------------------------------------------------------------

Now the real challenge is making the server syncable with clients in terms of the data being stored. Before what I was doing was making a Client Interface in the server, and the client implemented its own code, but this is bad because it quickly gets out of sync. I need to find a way for the server to store a client object as the client does, but the server might have more data associated with a client than the client does, so I've got some thinking to do.

Alternatively, I might be coupling game data and network design in a bad way, to which I need to think of an entirely different approach. Instead, I should be passing completed data to an outside class that is game specific and has nothing to do with the server/client code. I might try this approach since it would make the most sense in keeping the game specifics out of the network code and provide a reusable solution that does not tie the users code into my network code. I might make a thread later today asking about that.

## Conceptual Programming - Weak Objects and Interact

I've been trying to think of some "different" approaches to representing objects in classes. I began thinking about of the basic class members and functions that are normally implemented. For example, consider a simple player class that had such member variables as hitpoints, x, y, ammo, and function such as Move, Fire, Jump, etc... in those cases there is a strong mapping of variables and functions; they are explicitly defined. Interacting with the objects of that type just involves using the public functions. Pretty standard design.

So, with this new conceptual programming I am playing with things are setup like this: All objects derive from cEntity. The cEntity object has tow maps of values, strings and doubles. The cEntity class has API functions to Get, Set, Modfiy, or Remove attributes. The cEntity class also has private API functions that are invoked when those public API functions are used. Here is that class:

class cEntity
{
private:
CRITICAL_SECTION cs;
std::map classStrings;
std::mapdouble> classNumbers;

protected:
bool SetNumber(const std::string & name, double value);
bool RemoveNumber(const std::string & name);
bool ModifyNumber(const std::string & name, double value, double * outValue);
bool GetNumber(const std::string & name, double & outValue);

bool SetString(const std::string & name, std::string value);
bool RemoveString(const std::string & name);
bool GetString(const std::string & name, std::string & outValue);

virtual bool OnSetNumber(cEntity * entity, const std::string & name, double & value);
virtual bool OnRemoveNumber(cEntity * entity, const std::string & name);
virtual bool OnModifyNumber(cEntity * entity, const std::string & name, double value, double * outValue);
virtual bool OnGetNumber(cEntity * entity, const std::string & name, double & outValue);

virtual bool OnSetString(cEntity * entity, const std::string & name, std::string value);
virtual bool OnRemoveString(cEntity * entity, const std::string & name);
virtual bool OnGetString(cEntity * entity, const std::string & name, std::string & outValue);

public:
cEntity();
virtual ~cEntity();
cEntity(const cEntity & rhs);
cEntity & operator =(const cEntity & rhs);

bool SetNumber(cEntity * entity, const std::string & name, double value);
bool RemoveNumber(cEntity * entity, const std::string & name);
bool ModifyNumber(cEntity * entity, const std::string & name, double value, double * outValue);
bool GetNumber(cEntity * entity, const std::string & name, double & outValue);

bool SetString(cEntity * entity, const std::string & name, std::string value);
bool RemoveString(cEntity * entity, const std::string & name);
bool GetString(cEntity * entity, const std::string & name, std::string & outValue);

virtual void Interact(cEntity * entity);

virtual void Update(double delta);
};

Now this class is still being worked on but that is the basics of it all. The Interact function is an interface function that represents one entity interacting with another. The update would represent real time "processing" of all entities, but is not used yet.

Here is an example use of this setup.

#ifndef ENTITY_H_
#include "Entity.h"
#endif

//-----------------------------------------------------------------------------

class cBox : public cEntity
{
private:

public:
cBox()
{
SetString("name", "Box");
SetNumber("x", 0);
SetNumber("y", 0);
SetNumber("z", 0);
SetNumber("gravity", 1);
}
};

class cSpaceBox : public cEntity
{
private:

public:
cSpaceBox()
{
SetString("name", "SpaceBox");
SetNumber("x", 0);
SetNumber("y", 0);
SetNumber("z", 0);
}
};

class cImpossibleBox : public cEntity
{
private:

public:
cImpossibleBox()
{
SetString("name", "ImpossibleBox");
SetNumber("x", 0);
SetNumber("y", 0);
SetNumber("z", 0);
SetNumber("gravity", 1);
}

bool OnModifyNumber(cEntity * entity, const std::string & name, double value, double * outValue)
{
if(!entity || entity == this) return false;
if(name == "y")
{
std::string ename;
if(entity->GetString(this, "name", ename))
{
if(ename == "Gravity")
return false;
}
}
return true;
}
};

class cGravity : public cEntity
{
private:

public:
cGravity()
{
SetString("name", "Gravity");
SetNumber("g", -9.17);
}

void Interact(cEntity * entity)
{
double g = 0;
double y = 0;
double gravity = 0;
if(entity->GetNumber(this, "gravity", gravity))
{
if(entity->GetNumber(this, "y", y))
{
GetNumber("g", g);
printf("Position started at: %f\n", y);
if(entity->ModifyNumber(this, "y", g * gravity, &y))
{
entity->GetNumber(this, "y", y);
printf("Position is now at: %f\n", y);
}
else
{
printf("Entity denied the affects of gravity.\n");
}
}
else
{
printf("Entity has not properties that can be affected by gravity.\n");
}
}
else
{
printf("Entity cannot be affected by gravity.\n");
}
}
};

//-----------------------------------------------------------------------------

int main(int argc, char * argv[])
{
cGravity gravity;
cBox box1;
cSpaceBox box2;
cImpossibleBox box3;

gravity.Interact(&box1);
printf("\n");

gravity.Interact(&box2);
printf("\n");

gravity.Interact(&box3);
printf("\n");

return 0;
}

//-----------------------------------------------------------------------------

After running the program, the output is:

Position started at: 0.000000
Position is now at: -9.170000

Entity cannot be affected by gravity.

Position started at: 0.000000
Entity denied the affects of gravity.

Press any key to continue . . .

Whoa. Just imagine being able to have such control over any interactions in real life haha! Good bye gravity! Anyways, the design is meant to allow freedom and flexibility in terms of what an object represents as well as how it is modified. From a technological standpoint, anything would be possible as seen with preventing gravity. You could make gravity inversed so you actually float up or reduce/strengthen the effects of it. Of course, I am just using gravity as the prime example, any "force" would work the same, including things that we do not have, such as "side ways" gravity, which more or less would be a constant wind.

Moving on, I don't have any practical things to do with this concept, just experimenting. R&D so to speak. There are some problems that I've yet to really address. For example, let's assume I have my small universe of objects. Naturally due to how computers work, I will be handling Interactions and Updates in a sequential manner. Because of this, I do not have a way of implementing "order" or "priority" in the world. This would mean the implementation of the world is up to the actual program and not the class itself. I.e. one program might implement a number property that is a 1,2,3 and establishes a priority like that and another program might use a different system.

My idea was, if even possible, somehow have a game where players create their interactions with world objects. This is a bit vague so let me make up an example of what could be possible.

Start out with a cEntity that is a virtual tree. Let's say it has a number property of "wood" as 5000. Now, the player has their own cEntity that represents them. By intuition, the player Interacts with the tree and learns it has wood that is harvestable. For the sake of simplicity, let's say the player calls the ModifyNumber function of the tree with the "wood" property and a value of 1. The tree object verifies the operation in its OnModifyNumber and either allows the operation or denies it. Let's say it was successful and it then sets the wood property for the player to 1.

Not the best example and a lot of questions result from that, but that's the gist of it all. I think it'd be interesting if properly implemented, so I am going to keep playing with the idea until I get distracted by something else or make a breakthrough.

## They still don't match!

Thumbs Up/Down Icons Do Not Match!!

Don't blame me for being an OC...