Dangers of multi-threaded applications

Creating multi-threaded applications can be a scary task (at least for me). The common problems such as deadlocks, race conditions and corrupted memory are pretty intimidating, especially if you're just starting out. One less famous, but still nefarious, problem you can run into in multi-threaded development are continuously spinning threads. I'm sure there's a more common name for these, but essentially they are threads running in a non-stop loop, polling for new stuff to work on. Threads like these run as fast as possible in this loop, and greedily consume as much CPU time as they can; which leads to horrendous performance loss.

We ran into a very bad version of this today on new server software we mass tested for the first time. A lot of people complained about being unable to connect to the server, move through our game levels, or interact with items in the levels. I confirmed the server processes were running at 100% CPU, so we started looking at where our performance was going. What made matters frustrating was on our (beefy) dev machines, the servers were only running at about 8-10% usage. We spent a little time looking at some game engine profile output, and looking at some asset issues we thought were the culprit. In the end I wish I could say we found the problem through some tool or cool debug technique; but rather it came about when, on a whim, I asked another developer about some threaded functionality he recently added. I asked him to check his code to make sure his new thread was yielding if there was nothing for it to do, and voila, we found the problem.

It ended up he originally built the threaded task with some blocking functionality which had controlled the thread CPU usage (unintentionally), but later changed it to use non-blocking calls. After this change he forgot to add a thread sleep call, so the thread was simply spinning in a loop looking for something to do. While he fixed the code, I figured out how to setup a test case for our dev machines to reproduce the problem of 100% CPU usage on the servers. I remembered we could set the processor affinity through the task manager, which would allow us to simulate a single-core machine to run the game server on. Once I did this, I was able to see the game server spiking up to 100% CPU usage on the assigned core. Next I integrated our fix, which was to simply sleep the thread for 10ms every loop, and re-ran my test. The server dropped from 100% CPU usage to between 0 and 1% while idling!

Overall, it was a good day today, long as hell though.

Read an interesting Gamasutra blog today

I read an interesting blog on Gamasutra today written by an indie developer about his first week working full-time on his project. Reading it reminded me of what it was like for me when I first made the plunge from a mod-developer in my free time to a professional game developer full time. The biggest thing I remember thinking back then, was how awesome it was to have so much time available to work on projects. Going from 'doing what I can in my free time' to 'doing this for 8 hours a day' is quite a change. Some people handle that transition well, and some people don't.

As for me, I'm currently waiting to hear back from my project partner to see if he'll be available to meet up at a coffee place today to do some work. We've made some good progress on our project so far, and I've been pretty happy so far. I let one of my co-workers play an early version on my iPhone last night and he actually seemed to enjoy what was in there so far, so I'm pretty happy about that.

Good work day on Sunday

I had a very good working day on Sunday with my game development partner. On Thursday we agreed the scope of our previous project was too large for our tastes, and decided to go in a new direction. So far that decision has been a good one, the new idea we're running with is much smaller scope, and we're able to reuse a lot of content and code from the last project.

Last night I mocked up the movement for our main entity and it feels pretty good. The other cool thing we did was build and publish a test build to an iPhone and an Android device. So far there's no content or real gameplay yet, but it's looking much less daunting now.

Sweet Borderlands sale on Steam

Steam is currently hosting a pretty sweet sale for Borderlands and associated DLC content; for ~$20 you can grab the game and all 3 DLC packs.

I decided to pick it up seeing as one of my co-workers is currently playing Borderlands and has been trying to convince me to play. It looks pretty interesting, I've had my eye on it for awhile but never got around to trying it out because of other games and such. I'm still working as much as I can on my Unity3d project, it's coming along but not as fast as I would like it to.

On the server front, I haven't made any decisions yet. I still want to get a server of some sort, but I've been putting off on it for the past few weeks.

Looking at new server hardware

I was talking to my boss today as I drove him to the airport, and we got on the topic of home servers. I've been kicking around the idea of buying a server for awhile now, and now I'm really tempted to get one.

One of the biggest reasons I want a dedicated server box is so I can move my Perforce server off my desktop. For some reason I always hate having to leave my desktop running so I can have Perforce access when I'm away from home; it feels like such a waste. The idea of hosting my Perforce server on a dedicated server box really makes my nerd side happy :). In addition to that though, my boss pointed out I could run backups through that server as well, which is another big priority for me since I don't have any backup systems running for my home network.

I've been looking at the HP EX495 MediaSmart Home Server. It has some pretty good spec's on it, and the price is pretty good. Guess we'll see what happens

Update on things

Been pretty busy lately. Had to travel to a conference for work in the beginning of May, been bouncing around different projects at work, and my buddy and I have been putting in a lot of time on our new game idea.

Things at work have been especially interesting the past week when I got moved over back fill on a project for my boss. We have a piece of software we're upgrading and preparing to license; however while the functionality the software provides is pretty good, the underlying code is pretty messy and old. The past week I've been getting oriented in this new code-base, and doing a lot of clean-ups and improvements. The MS C++ compiler (VS 2008) has been pretty handy in this regard, it spits out a lot of good info regarding deprecated or dangerous functionality. Working on this has been an interesting diversion and challenge, so I'm pretty happy doing it.

iPad release day today

Apple's new iPad comes out today, that's pretty cool. I've been debating the pros and cons of getting one myself; and for now I'm leaning to not getting one for the immediate future. I'd love to have an iPad to play with and to develop games on, however right now those 2 reasons aren't quite enough to justify dropping $5-600. I'm still working on an iPhone project, and the iPad would easily distract me because I'd want to play with it.

Speaking of which, Unity3D just came out with an update to their iPhone SDK, Unity iPhone 1.7. I'm pretty stoked about that update, especially being able to play your game in the iPad simulator. That'll let developers without an iPad get a feeling for how their game will look running on the iPad device (not a perfect example though). I already downloaded and installed the update, however now I'm waiting to finish downloading the most recent iPhone/iPad SDK from Apple. Once that's finished and installed, I'm going to rebuild my current test project and upload it to my phone to check for problems. The Unity folks do a fantastic job with their updates, but every now and then a problem slips in, so it's always good practice to do some regression tests.

Annoyances with C#'s DirectoryInfo.Delete()

Taking a small break from writing up a game design doc for my iPhone game. It's interesting being the one actually writing a doc for a game, rather than reading one made by someone else. It does make me appreciate the amount of thought and work our designers put in.

Anyway, last week at work I was writing a new C# console utility to automate directory clean-up tasks on our build server. Things went pretty well until I hit a snag with DirectoryInfo.Delete(Boolean) not working properly and throwing a constant stream of exceptions. It ends up it was error'ing out because the directories I was trying to delete (which contain a good number of sub-directories of arbitrary depths, and several thousand files), contained a number of read-only files. After looking around a bit, it appears this is intended behavior, as it follows the idea of the pop-up notification users get when they try to manually delete directories.

To get around this, I ended up creating a simple method to iterate through all directories and files under a particular directory, and set all their Attributes to normal. Sorry for the formatting.

// Set attributes on all dirs and files in the passed in directory to 'Normal'
void SetDirectoryContentsToNormal ( DirectoryInfo Dir )
{
if ( Dir.Exists )
{
Dir.Attributes = FileAttributes.Normal;
DirectoryInfo[] SubDirs = Dir.GetDirectories("*", SearchOption.AllDirectories);

foreach ( DirectoryInfo SubDir in SubDirs )
{
SubDir.Attributes = FileAttributes.Normal;
}

FileInfo[] FilesInDir = Dir.GetFiles("*", SearchOption.AllDirectories);

foreach ( FileInfo file in FilesInDir )
{
file.Attributes = FileAttributes.Normal;
}
}
}

This does cause a little pause when running, however for our particular circumstance it's perfectly fine because the tool runs only once a day in the early morning.

2010 Census

Finally got around to filling out my 2010 Census. If you received one make sure to fill it out as well. I will say I was surprised how short it was, I thought it would have a lot more questions. There are so many instructions and help hints and clarifications on the form I expected it to be more complex, I thought I did something wrong when I finished in about 10 minutes. Funnily enough, there was a note at the bottom of the form saying on average it would take about 10 minutes.

Thinking about it though, I can see why they had that much help embedded on the form. The Census is done by the Gov. to help determine the distribution of the population; and the results are used as the basis for policy decisions, budgetary allocations and government representation for the next decade. Making sure it's as clear and precise as possible is extremely important.

I do wonder how many iterations, focus tests, expert reviews and God knows what else was done to test the usability of the final Census form. Just the fact they had the 10 minute estimation means some of their testers had the same anxiety as I did...

Looking at website stuff

Finally decided I need to start learning more about web technologies. This area has been a big hole in my field of knowledge, and I figured I need to address that. First thing first has been figuring out WHAT web technology to learn, there are quite a few options out there and narrowing them down is a little daunting.

I'm leaning heavily toward ASP.NET, as I'm passingly familiar with the .NET framework now and feel comfortable working in that sandbox. There's also the huge benefit of being able to use Visual Web Developer IDE, which at first glance appears extremely powerful and well documented. My boss has worked on web stuff on and off for the past 10 years or so, and original came from a php background. He's been rebuilding his website with ASP.NET MVC, and he's enjoying it quite a bit. Personally too, I really don't feel like learning php/javascript all that much. I could learn them, but since I'm currently working on 2 other projects right now, I really don't have a ton of extra time to spare learning completely new languages. Being able to work in libraries and frameworks I'm already working in (.NET) is a pretty big draw.

The next step is actually finding a hosting plan...

Annoying distractions

I was working on my iPhone game on Sunday, and wanted to load the latest build onto the device so I could test some stuff out. When the build came up on the screen, the main menu looked suspiciously like the old version of the game, and didn't contain the new test map I wanted to try out. I figured I did something wrong so I went back to XCode to 'Clean' and 'Build' my executable again...and received an Error in the output.

It ends up my iPhone developer certificate had just expired. In hindsight, I'm very glad this issue came up when it did. Since it was a Sunday I had the extra time to renew my iPhone dev certificate, and because the change I wanted to test was obviously not there, finding the underlying issues didn't take very long. It was an annoyance though. I ended up wasting about 20 minutes getting my stuff back in order and at the point I could build against my device again.

Oh well, at least I don't need to worry about this again until next year.

Optimization fun with new project

I'm working on an iPhone project with a buddy, and the progress has been slow-ish going so far. Our biggest problem is committing to an idea; we have a good number of them, however we haven't found a gameplay combination we're fully ready to jump on. I feel like we're getting closer though, we've done a good job of bouncing new ideas around and trying to prototype stuff out.

While we're figuring out gameplay, I'm working on creating some generic utilities to assist us in development and iteration. One item that comes to mind is a search method I wrote several weeks ago. It provides a nice API method for finding a specific object by a name within a hierarchy of objects in the game scene. This method (call it FindGameObject) worked fairly well, however I found some limitations to it which I knew would cause a slowdown on the iPhone given the number unnecessary operations I was doing. Tonight I decided to trim this down to make it leaner and meaner with some ideas I came up with.

First thing I did was write a test script to provide profiling support and get a baseline of how much time it took to run 10 million searches using FindGameObject on a deep heirarchy (about 8 objects). I decided on 10 million somewhat randomly, however it was deliberately a large number to enhance the visible impact of small changes. Then I wrote the new version of my method (call it FindGameObject2) with my new changes and hooked it into the test script. The results were:

FindGameObject - Took: 6.948123 sec to run 1000000 lookups
FindGameObject2 - Took: 5.916583 sec to run 1000000 lookups

This is a rough average of about 5 runs, not a scientifically acceptable test set but good enough for our purposes. Comparing against the current functionality, you can see I gain ~1 second when doing 10 million lookups, which is about a 15% increase

Before I finish I must mention these tests were run on my PC dev machine, not the actual iPhone hardware. This makes it especially important to possess a test case I can reliably run on the iPhone hardware, so I can compare my changes against the original functionality on the target hardware. However I expect my optimizations to yield a comparable speed increase on the iPhone hardware.

Computer security scare

When I was at a friends house last weekend working on an iPhone project, I had a big computer security scare. We'd reached a point where I wanted to transfer a build over him so he could play around with it, so I put the build in my network share folder. When he browsed to my computer though, instead of just seeing my read-only drop box, he saw my entire programming directory with ~10 projects in it...and he had read/write access to it! Let me say that again, my main development folder had read/write access privs set for everyone, essentially it was a shared public folder

Now this is a pretty stupid thing to set, and what makes me kick myself even more is I'd set this many many times, not knowing the implications of what I was doing. I'd exposed my precious data to anyone to read or even modify. This was especially scary because I like working at coffee shops, so I was very lucky some bored, tech-savvy person didn't go and delete the projects I'd spent hours or days on.

In case you're wondering why I'd set public read/write privs on a folder that's so important, the answer is it was a workflow I was using for development. I had to share symlinks between my OS X and WinXP installations (I run WinXP through a virtual machine). There are some nuances regarding sharing symlinks between these two OS's (I don't recall the exact issue), but a workaround I found was to, among other things, publically share my dev folder. What really kills me though, is I found a better workflow several months ago that didn't require symlinks at all.

Moral of the story, make sure your important data directories aren't set to be shared publicly :)

GDI leak hunting season

In addition to my create an installer task, the past several days I've been tasked with finding an extremely nasty bug in our project. The game would run fine for several minutes, however inevitably it would begin to exhibit some strange behavior, such as random text strings no longer rendering or the entire game window, or even monitor display, flickering wildly. Obviously QA would never allow us to ship a product in such a condition, so I had to dive in and figure out what was going on.

To start off, I've previously never delved very deeply into the Windows API. The beautiful thing about working with an existing game engine is most of the low-level OS interactions are abstracted away, so you only need to worry about implementing new, cool features on top of that layer. Unfortunately for this project, we needed to do some very low level interaction with the Windows API in order to fulfill a feature request by our client. In this case, we were interacting with the Windows Graphics Device Interface API to handle some custom font rendering. We made some good initial progress on the feature implementation, however the programmer working on it had to suddenly take some personal leave time for family reasons. This left us in a lurch because the feature implementation, while working as a prototype, wasn't fully completed, cleaned-up and optimized yet. One of the nasty caveats with Windows GDI Object handles is there are a limit number of handles you can have active (~10000 on my XP system). Once you reach that number, the OS appears to start doing some very odd things, which causes things to disappear and applications to start flickering.

I'll save you the boredom of describing my late nights looking over code and reading GDI documentation, but suffice to say it was a late couple nights :). Thinking back now, I could kick myself for not doing some obvious things until today. My breakthrough today came when I finally looked up online some additional resources about GDI memory leaks and how to find them. There are some interesting articles and techniques for finding GDI leaks, however I felt they were all overkill for our particular situation. Most of those techniques were for finding a leak *somewhere*, however in our case I knew we had a leak in a very specific area of our code (about 400 lines altogether). The real breakthrough came when I found out you can add a 'GDI Object' column in the Windows Task Manager (View->Select Columns), and watch in real time as the count goes up and down for a particular application. On the surface this doesn't appear incredibly useful finding a leak *somewhere*, however as I said above, I knew the leak was happening in a very specific area. So I fired up the trusty debugger and began stepping through the suspect code line-by-line while watching the GDI Object count; and after several minutes I found the offending item. A CreateSolidBrush() call hidden deep within an if block was allocating a new handle, but never deleting it. After I fixed the leaking object I ran the game again and confirmed our GDI Object count was now steady.

Victory!

Installer purgatory

I spent most of the day yesterday creating an installer for one of our projects. Unfortunately, I wasn't told we needed an installer until pretty late in the project, so I had to drop everything and make that my number 1 priority. And by pretty late in the project, I mean I had about one-and-a-half days to make it.

Installers. They've become one of my arch-enemy tasks at work. That part of my job is a poster-child for why you shouldn't volunteer to learn something, because you instantly become the person who always does it. Scott Adams occasionally points this out in his dilbert cartoon strip...if only Scott's warning had reached me sooner!

The biggest reason I hate working on installers is they're generally very tedious and time-consuming to create and test. An installer is the user's first impression of your game, and if it fails to work properly, they will have a negative impression of your product before they even get a chance to play it. Consequently, installers must be rock-solid and bug free on all your supported OS's, otherwise you could face some real nightmare scenarios, such as this or this.

Snowed in with Mass Effect 2

Big snow storm hit the East Coast this weekend, so I stayed in the entire weekend playing Mass Effect 2.

Let me say I loved that game, I ended up putting in ~40 hours on my first run through as a Sentinel. Overall I enjoyed the Sentinel gameplay, it gave a good mix of tech and biotic powers that allowed a lot of flexibility when faced with a wide variety of foes. Because I was able to deal with organics, mechs and shield/armor based foes myself, I had a good amount of latitude in my teammate choices. One thing I hate is feeling forced into using certain people just because you have to cover a hole in your offensive capabilities.

I'm probably gonna play through again as a Soldier to get a feel for the game in a different play style, although I'm also pondering playing as a Vanguard. One of my co-workers played as a Vanguard and wasn't as happy with it because he felt it was a one-trick pony with biotic charge. Once you charged, you were left in the middle of a group of enemies without any cover and gimped weapons (he didn't think the shotgun was very powerful, even fully upgraded).

From a game developer point of view, I was very impressed with the game. I saw a number of small bugs, like the camera not focusing correctly or characters not appearing in the camera's view; plus a couple of show-stopper bugs like getting stuck on invisible walls. I had 2 instances where I was forced to restart the current mission because I was stuck on a wall and couldn't move. The wide variety of possible endings was another thing I found very cool. Talking to my co-workers today, the four of us who finished the game over the weekend all experienced slightly different endings. Providing the player with the ability to influence the ending like that was a very very cool move on Bioware's part, and it would be interesting to see that in more games.