Conference Overview

This year's online gaming conference LOGIN was held once again at the Seattle Marriott Waterfront hotel, which is right on the shore of Puget Sound a hop skip and a jump away from downtown Seattle. LOGIN has been around for three years, however it's been through a few names in its time. Originally called OGDC for Online Game Developers Conference, they were smacked down by CMP (now Think Services) for incorporating the "GDC" trademark into their conference name. That may ring some bells - we'll get to that in just a moment. The following year, they named themselves ION and once again faced litigation, this time from some random dude claiming to own the trademark for "ION". While the name LOGIN hasn't given them any issues this time around (and will hopefully be the name next year, although the running joke among recurring attendees is "what will it be called next year?") they were partners with Howell Expo, which organizes the China Game Developers Conference. Yea - CGDC. So the litigation hit them again in a rather roundabout way.

Still, for all the changes that the conference identity has gone through, its content and focus as an event has always been of top notch, and none of the attendees have ever particularly cared what it was called, so long as it delivered with quality sessions, bountiful networking opportunities, and copious amounts of yummalicious food. And it did. Many of the comments I heard from other attendees I spoke to during the conference were in praise of those three attributes.

The Sessions

For a relatively small conference, LOGIN packs a lot of punch. From sun up to sun down there's something going on, and while most conferences that last 2-3 days will have two or three tracks, LOGIN had six - albeit not all were evenly represented. Still, 4 sessions would be going on at any one time, with a total of 6 each day, except the last which had 5. The tracks were: Legal, Business, Design, Community, Globalization, and Programming/Tech. One of the things I thought was most ingenious at LOGIN since its inception that I have yet to witness at any other conference is the lunch keynote. The conference is small enough to fit all attendees into a ballroom for lunch. After 30 minutes of socializing and eating the keynote speaker takes the stage for an hour-long talk. During so however you can still get up and go for seconds (or thirds) or munch on some dessert. It makes bad keynotes tolerable. Well, I guess I can't really classify any LOGIN keynote as "bad". Uninteresting would be a better word - and that's just my personal opinion on the ones in question over the years.

This year they had an opening session the day before the conference really got started, where all the speakers got to come on stage for a minute or so and pitch their talk to the assembled conference attendees. I liked the idea, but it didn't seem that well executed and some speakers didn't seem to fully grasp the concept while others just weren't able to make it that early to take the stage themselves. That's not to say it still wasn't useful. I listened to the pitches and rated all the talks I thought sounded interesting. It did help me a little bit in deciding where I wanted to go sometimes. It would have benefited though from more speaker participation and understanding, and a hard cutoff limit to keep the ramblers at bay.

Another new session this year was the debates. Each day had a debate focused around a topic that would have two sides, a For and Against, duking it out with each other over the course of an hour. The idea wasn't really to come to any sort of conclusion or crown one side victorious over another, but to take a topic and break down its various viewpoints from developers close to the subject matter at hand. How did it go? I'm sorry to say I didn't attend any. Perhaps from a pure editorial perspective I should have at least sat in on one, but all were scheduled against sessions that actually interested me. In retrospect, I could have missed the session I sat in on Wednesday.

The Networking

One of the best things about LOGIN, to me, is the ability to see everyone I know who is going to be there at some point - usually multiple times. For extended periods of time as well. The thing about larger events like GDC is that some of the developers I know I spot in the hall or out on the street as I transition from one session to another. All we have time to do is give each other a quick wave as we continue on our way. That might be all I see of them the rest of the week. At LOGIN however it's a different story, as everyone's in the same place all day.

LOGIN also has a nice 30 minute buffer between sessions throughout the day, which makes the whole event slow down to a nice leisurely pace. It also means you can hold a meaningful conversation with someone between sessions. Or multiple people, even.

Parties are a required part of any conference, and LOGIN always delivers. This year they had an opening reception, an official conference party (unfortunately not as swank as last years party at the Seattle Aquarium, though Joe the Magician helped make up for it) and the Seattle IGDA chapter also held an after-conference event on the last day.

Every morning, a networking breakfast is held for those willing and/or able to crawl out of bed at 7:30am. I did it once this year and once last year. Lissa somehow managed to do it twice. Each breakfast is handled speed-dating style, and every few minutes you're sent to another table via some means - like my breakfast used playing cards and people were shuffled around based on odd/even numbers and suits. If you're able to wake up in time, they're worth the effort. And not just for the potential business - which brings us to our last LOGIN attribute...

The Food

Does this really deserve a section all to itself? Oh yes, it does. I like food. Don't you like food? Food is good, and it's amazing how hungry you can get just sitting on your ass all day listening to people speak. Seriously.

If you've been to GDC, you know well the infamous boxed lunches. They aren't bad, but they're nothing great either. MIGS has a cafeteria on-site and they give you lunch vouchers, but it's still cafeteria food. Austin GDC has a ton of great food outside the conference and in Austin proper (BBQ FTW!), but you still have to leave the conference center. Same for IGC East up in Boston.

LOGIN is different. Last year, they served sushi one day. This year one of the lunch offerings was wood-grilled salmon. We're talking full-on gourmet meals here. Breakfast has a full helping of pastries, bagels, french toast, eggs, etc. The parties all have decent finger food, the networking party had huge shrimp and crab legs.

Glorious food! Bountiful food!

My Nitpicks

Nothing is ever perfect however. While the subject of food is still fresh, let me say that last year was better in regards to the snacks you received in between sessions, which ranged from ice cream bars to health bars. They were noticeably absent this year.

There was no conference wireless, which crippled my hopes of live tweeting sessions again. The second day I discovered that the Marriott hotel access point actually offered free wireless access, but it was very spotty down in the ballroom level, which was below the hotel lobby. A dedicated conference access point was available, but not enabled for us this year. It was rather ironic to me that an online games conference wasn't online.

There was no soda. It made me wish I were a fan of water, iced tea or coffee. Luckily the hotel lobby cafe had a nicely-stocked fridge.

Next Year

Will any of the above minor inconveniences affect my decision to attend next year? Of course not. Hopefully LOGIN 2010 (or whatever it's called) will be back once again. In addition, hopefully it and GDC Canada will play more nicely instead of stepping all over each other like they did this year. Next year GDC Canada is scheduled to run May 6-7. Given the short nature of both conferences, a LOGIN event from May 3-5 would complete the entire business week. Also given the short travel distance between Vancouver and Seattle (a little over 3 hours by bus) there could be some serious cross-pollination between the two events. Another option for LOGIN would be the following week, May 10-13, giving people the weekend to travel. Either one would be awesome. I have my fingers crossed.

actually I wonder if I would actually survive attending and then covering three conferences in a row.

Select Session Coverage

Designing with Cheaters in Mind

Brian Green (Near Death Studios, Inc.)

This isn't about how to design your game to prevent cheating. You can't do that. This is about designing your game to mitigate cheating. It's also not designing in terms of gameplay, but in terms of how your game system functions - how all the pieces interact. That's where a lot of player cheating will stem from: finding flaws in the system.

What is cheating? It's interacting with a game to gain an advantage that others cannot. These run the gamut from simply not following the rules of the game (don't create multiple accounts), running exploits (scripts), or abusing design flaws (a certain armor piece remains in affect even when removed). Why should you work so hard to counteract cheating? Because it's simply not fair and will cause players to not want to play your game, and it's just bad practice to leave harmful system effects unfixed that players could use to cheat or potentially even use to hack into your game or other people's machines.

While the design of the game can be the first place to stop cheating from occurring, it won't solve everything. Programmers can also fix issues however they can also create their own as well. Customer Service will see the brunt of any affects cheating has on players, so they're your front line of defense, but Community Managers are your early warning system, as they will be able to spot trends in the community that could lead to the discovery of players covertly cheating.

Why do players cheat? Dr. Richard Bartle has 4 basic types of cheaters:

• Achievers - these people are looking for quick recognition. They want to be on the top of the heap as fast as possible and stay there
• Explorers - not always cheaters per se, they do exploit glitches to get to areas of the world they're not supposed to be
• Socializers - usually looking for payback in some form or another for being socially rejected
• Killers - these players want to dominate others, and usually exploit cheats that let them kill swiftly, or widely

There are also meta-game reasons that people would want to cheat, such as simply sticking it to "the man" (the game creators/admins), getting access to extra resources to sell for real-world money, or consequence-free behavior - such as hacking into someone else's account to use for a PvP killing spree.

The best time to catch possible cheats is before game launch, preferably before beta testing. Just as with any other change you would make during a project, the sooner you do it he less it will cost. You also don't have to worry about any backwards compatibility or masses of players to piss off by taking some part of the game offline to fix it. However it's easy to miss cheats during this time because 1) developers who playtest don't usually try to cheat 2) you don't have a lot of people playtesting the game and 3) some play testers may actually try to hide a cheat until game launch when they can exploit it. Once a game is live you have thousands of players - the numbers will always be against you.

Catching cheats when the game is launched requires special handling. You have to fix the problem quickly, but not so quickly that you push out a hotfix that only makes the problem worse, or fixes that cheat and introduces a new vulnerability. Clear communication between the Community Manager and the community is key to letting players know action is being taken but time is also being taken to ensure the fix is fully effective. Make sure your Customer Support knows the status of the fix as well so they can inform players calling in for support if you've taken part of the game temporarily offline. Programmers and designers need to be talking to ensure that any fix to the game does not upset the game balance or game play. The whole team has to be mobilized to deal with this.

Detecting cheats quickly can be achieved by a number of means. One obvious way is to have an efficient Customer Support department and Community Manager, so that any complaints from players who have witnessed cheating are handled promptly. A better way however is to actively monitor the community and game to try and discover improper behavior before it becomes a major problem. This can be done with the proper community management tools (flagging system, user activity tracking, etc) and proper game metrics (server logs, client logs, player stat tracking, etc). Study previous games to find out how players cheated to get an idea of what to look for in the data. If you detect a cheat, don't always swoop in right away to fix it. It's one thing to discover a cheat, it's another to actually catch a person(s) cheating.

Top ten lessons:

1. Listen to players about possible cheats - then get specific info from your team and metrics system to confirm suspicions
2. Use mod value for temporary changes - don't modify the base value of an item or game setting, instead replace it with a temporary value so you can switch back easily if things go bad
3. Make on the fly changes easy - build your game to allow for quick and easy changes so you can react to cheats faster
4. Don't put test items on a live server - oops? It happens. Why would you have a one-shot kill item anyways?
5. Don't ignore social engineering - people will believe anything, someone will let a cheat or even an imagined one get to their head
6. Don't let your ego get control - people are out there that are smarter than you. Accept it and accept the fact that they will find ways around your system
7. Don't be afraid to punish cheaters - legit players will start to lose faith if they see repeat offenders only getting a light rap on the hand, on the other hand players will respect the game rules more if the law is laid down. Ex: zeroing of Gamer Points
8. Competition (PvP) makes things worse - be ready for more problems if you have PvP in your games - players hate unfair advantages and they come about often when people cheat
9. Be realistic about cheats - people will find cheats, be ready for it
10. Just keep the game fun - don't punish for the sake of punishment, stop cheaters to make the game fun for others

Pathfinding Solutions for Large Online Worlds

John W. Ratcliff, Simutronics Corporation

I'll admit this is something I never really thought about myself until now - the fact that in an online game, NPCs have a lot of terrain to navigate over. And they're only getting larger and larger as technology progresses. The usual pathfinding methods don't work as well as they do in single player games, mainly because of the fact that MMO games are mainly server-based and share a lot of resources. The algorithms also don't scale well to the large open spaces in MMOs versus the small, tight FPS levels and perform slowly. MMOs are also dynamic worlds hosting thousands of players, so not only do you have to account for changing environments but thousands of pathing requests per second. And it all has to be extremely stable, lest you bring down part or all of the server and piss off thousands of paying customers.

A typical solution is to generate nav meshes, but John has surprisingly found that a more common word is "create". He was appalled to learn that the majority of MMO nav meshes were done by hand in an editor. This is a bad idea for the aforementioned reason of changing environments. MMOs are not static, and neither should the nav mesh be. These meshes are a layer visible to NPCs that help to guide them along paths and areas were walking, swimming, jumping or flying is allowed. Edges connect to nodes, and characters are directed along the edges from node to node to create a path. It's an expert system that is fully aware of the lay of the land so it also describes the connectivity between nodes in the mesh, for example whether an NPC can jump from one node to another. Interpolating the resulting path can smooth it out so the character's walk looks natural. The nav mesh can also be used to store meta data, such as where NPCs spawn, patrol areas, custom regions and programmable triggers. You also have to make sure to account for various-sized NPCs, either by storing such data in the mesh or creating additional mesh layers. To keep required data to a minimum John recommends classifying NPCs as either small, medium or large. Latency must also be accounted for, as in the time it takes for a path to be created, the character may have moved beyond the starting node, causing it to turn around, walk back to the start point and then turn around again to continue on the path. This obviously looks stupid so allow characters to blend the difference.

John went over several middleware, after admitting how useless he found it to try and roll his own system.

• Kynapse: Didn't have an API. At all. This was very crippling to its use, and also had long computational times
• Havok AI: John found this middleware interesting but was unable to share any data on it. He recommends a look
• PathEngine: This middleware also returned long times in computing paths

The one middleware technology John found most promising was from NavPower, and the technique is also being experimented on by Miko Mononen in Recast and Detour. This link requires a premium membership to AIGameDev.com, but you can download slides from NavPower's site here that explain the technique. AIGameDev.com describes it as "based on rasterization and voxelization, which treat space as a discrete grid and post-process these voxels to retrieve polygons on the output." This produces extremely fast results and is a very robust solution as well. It does still have its problems though that still need working out. John found that it doesn't do so well on stairs and jumping up and down off of objects. In these cases you may end up using a navigation mesh for such finer movements.

John's other reference was the AI Wisdom 4 article on auto generation of navigation meshes.

What you already know about building secure software...

Dave Weinstein, Microsoft

Dave Weinstein is good at his job, and that job is to be paranoid. Of everything. Well okay, I'm sure he doesn't think the coffee machine in is office is planning worldwide domination, and perhaps he's not constantly worried about his house being broken into. But he is on the constant lookout for hackers trying to worm their way into whatever system he is tasked to make sure is secure - and that means not trusting a single bit of data that system receives from a remote client. Likewise, he expects that whatever remote server he connects to doesn't trust a single bit of data that he sends it.

Sounds like a quandary? If both parties can't trust each other, how do they communicate? Very, very carefully.

Hackers aren't motivated by what drives normal people to play games. They simply want to gain some level of control over your system and use it for their own devious means. This sole purpose will drive them to formulate all sorts of attacks, and you can bet that you won't think of them all. So if you can't prevent someone from breaking into your server or hijacking your client then the best you can do is make it as difficult as possible for them, and hope they simply give up and move on to some other, more vulnerable game. Of course, there's also the mentality among hackers to be the first one to beat the system, and that if a suitable challenge presents itself they will work tirelessly at figuring out a way to solve it. That's how things like Blu-ray encryption get broken.

So okay, either way you can't really win. Build that bunker!!

The problem is that, while many people do mistrust data in online games, they usually don't distrust enough. The first thing that security people always look for is corrupt packet data. Older games that performed a lot of client-side calculations fell victim to this. Air Warrior, which ran all its flight models client-side and did not validate on the server, quickly saw players flying higher and faster than they were supposed to be able to. Other client-side data that can be manipulated if it's available to a client can be position, class/level info, authoritative combat, etc. Note that this is all packet data that were looking at here, but actually the entire packet should be considered a security risk, as hackers can modify the size, format and sender (UDP injection - which affects many online games that take advantage of UDP vs. TCP) to break through to your system. This is where you should apply a technique known as fuzzing to see if your system is susceptible to these forms of attacks. You should also review features that can be unsafe by design such as a tech-support ability to run commands on a user's machine or an auto-update feature that could be hijacked to send malicious updates to clients.

So you shouldn't trust you clients. But your clients can trust your servers right? After all, they're usually directly under your control. The client connects to the server and sends a verification code. The server goes okay! and you're all good right? Uhm, are you sure you're actually connecting to the game server? Internet connections are generally done by hostname, and just asking for server.game.publisher.com doesn't guarantee you a connection with the actual game server. First, there's DNS poisoning. Then there's also ARP impersonation on a local network. How to hackers get on your local network? Well it turns out that wireless routers in the same area can have the same SSID and password. Are you sure you're connecting to your router when you log on to the internet? Connecting to a rogue server can allow inappropriate content to be streamed into your game, or even allow the game to be patched with malicious code as Evilgrade demonstrated. So clients can't trust servers either, and mutual authentication is required.

Another threat is posed by scripting systems. Lots of games have them to allow their users to create new scenarios or objects, and lots of hackers love to exploit them to run programs that give them and advantage or control over some part of the game. Usually these problems manifest themselves after the game has been out in the wild for a while, but a lot of vulnerabilities can be found via an audit by a security professional. These are people whose sole purpose is to pour over your code and try to break it. They can be well worth the additional cost to ensure that your scripting system (and other systems) are secure so that your players can get full enjoyment out of extending their game play experience. This should also be something that is actually planned into the development cycle, not a last-minute decision that will cost you tons of money if anything needs to be fixed/overhauled/cut.

Finally, don't forget the data that you store on people's machines. Media formats are complex, large and in binary - hackers like this. Anything stored as data can be interpreted as code. While games usually include custom complex formats with accompanying custom parsers in the game to read them, humans have a distressing tendency to write vulnerable parsers - especially if they control both the tools that produce the content and products that consume the format. People always think it's good enough for them - but there's always someone better than you. Some examples include replacing camouflage colors in a texture with high-contrast colors - now everything is extremely visible. Or increasing the volume of footsteps so the player can't be snuck up on. Or changing model data to alter the hitbox. This is where fuzzing comes back into play again. Fuzz your parsers, and your signature validation code for the client/server.

The hackers are out there, and they're gunning for you because you're a new system that hasn't been tested yet and they all want first dibs. Fuzz all exposed data surfaces, any user provided content and network traffic. If you can't hire a security expert to audit your code, it may be best to simply not expose code to the player through a scripting language. Security experts like to say "think like an attacker" but that doesn't just apply to how an attacker will attack your system, but why.

Building Like You Play: The Mechanics of a 100% Virtual Development Team

Ed Note: I was planning on detailing a fourth session as well but, after talking to the presenter, I discovered the article he based his talk on that I had seen in Casual Connect magazine was still in his possession. We are currently working to bring that paper over to GameDev.net as a Featured Article. So look for it in a few weeks!