Creating multi-threaded applications can be a scary task (at least for me). The common problems such as deadlocks, race conditions and corrupted memory are pretty intimidating, especially if you're just starting out. One less famous, but still nefarious, problem you can run into in multi-threaded development are continuously spinning threads. I'm sure there's a more common name for these, but essentially they are threads running in a non-stop loop, polling for new stuff to work on. Threads like these run as fast as possible in this loop, and greedily consume as much CPU time as they can; which leads to horrendous performance loss.
We ran into a very bad version of this today on new server software we mass tested for the first time. A lot of people complained about being unable to connect to the server, move through our game levels, or interact with items in the levels. I confirmed the server processes were running at 100% CPU, so we started looking at where our performance was going. What made matters frustrating was on our (beefy) dev machines, the servers were only running at about 8-10% usage. We spent a little time looking at some game engine profile output, and looking at some asset issues we thought were the culprit. In the end I wish I could say we found the problem through some tool or cool debug technique; but rather it came about when, on a whim, I asked another developer about some threaded functionality he recently added. I asked him to check his code to make sure his new thread was yielding if there was nothing for it to do, and voila, we found the problem.
It ended up he originally built the threaded task with some blocking functionality which had controlled the thread CPU usage (unintentionally), but later changed it to use non-blocking calls. After this change he forgot to add a thread sleep call, so the thread was simply spinning in a loop looking for something to do. While he fixed the code, I figured out how to setup a test case for our dev machines to reproduce the problem of 100% CPU usage on the servers. I remembered we could set the processor affinity through the task manager, which would allow us to simulate a single-core machine to run the game server on. Once I did this, I was able to see the game server spiking up to 100% CPU usage on the assigned core. Next I integrated our fix, which was to simply sleep the thread for 10ms every loop, and re-ran my test. The server dropped from 100% CPU usage to between 0 and 1% while idling!
Overall, it was a good day today, long as hell though.
I'm the co-founder and programmer of Nakai Entertainment (www.NakaiEntertainment.com), a small mobile games company in Raleigh, NC. I talk a lot about programming, game development and design, and games engines like Unity3d and Unreal.
Friday, November 12, 2010
Sunday, September 19, 2010
Read an interesting Gamasutra blog today
I read an interesting blog on Gamasutra today written by an indie developer about his first week working full-time on his project. Reading it reminded me of what it was like for me when I first made the plunge from a mod-developer in my free time to a professional game developer full time. The biggest thing I remember thinking back then, was how awesome it was to have so much time available to work on projects. Going from 'doing what I can in my free time' to 'doing this for 8 hours a day' is quite a change. Some people handle that transition well, and some people don't.
As for me, I'm currently waiting to hear back from my project partner to see if he'll be available to meet up at a coffee place today to do some work. We've made some good progress on our project so far, and I've been pretty happy so far. I let one of my co-workers play an early version on my iPhone last night and he actually seemed to enjoy what was in there so far, so I'm pretty happy about that.
As for me, I'm currently waiting to hear back from my project partner to see if he'll be available to meet up at a coffee place today to do some work. We've made some good progress on our project so far, and I've been pretty happy so far. I let one of my co-workers play an early version on my iPhone last night and he actually seemed to enjoy what was in there so far, so I'm pretty happy about that.
Monday, August 30, 2010
Good work day on Sunday
I had a very good working day on Sunday with my game development partner. On Thursday we agreed the scope of our previous project was too large for our tastes, and decided to go in a new direction. So far that decision has been a good one, the new idea we're running with is much smaller scope, and we're able to reuse a lot of content and code from the last project.
Last night I mocked up the movement for our main entity and it feels pretty good. The other cool thing we did was build and publish a test build to an iPhone and an Android device. So far there's no content or real gameplay yet, but it's looking much less daunting now.
Last night I mocked up the movement for our main entity and it feels pretty good. The other cool thing we did was build and publish a test build to an iPhone and an Android device. So far there's no content or real gameplay yet, but it's looking much less daunting now.
Tuesday, August 17, 2010
Sweet Borderlands sale on Steam
Steam is currently hosting a pretty sweet sale for Borderlands and associated DLC content; for ~$20 you can grab the game and all 3 DLC packs.
I decided to pick it up seeing as one of my co-workers is currently playing Borderlands and has been trying to convince me to play. It looks pretty interesting, I've had my eye on it for awhile but never got around to trying it out because of other games and such. I'm still working as much as I can on my Unity3d project, it's coming along but not as fast as I would like it to.
On the server front, I haven't made any decisions yet. I still want to get a server of some sort, but I've been putting off on it for the past few weeks.
I decided to pick it up seeing as one of my co-workers is currently playing Borderlands and has been trying to convince me to play. It looks pretty interesting, I've had my eye on it for awhile but never got around to trying it out because of other games and such. I'm still working as much as I can on my Unity3d project, it's coming along but not as fast as I would like it to.
On the server front, I haven't made any decisions yet. I still want to get a server of some sort, but I've been putting off on it for the past few weeks.
Thursday, August 5, 2010
Looking at new server hardware
I was talking to my boss today as I drove him to the airport, and we got on the topic of home servers. I've been kicking around the idea of buying a server for awhile now, and now I'm really tempted to get one.
One of the biggest reasons I want a dedicated server box is so I can move my Perforce server off my desktop. For some reason I always hate having to leave my desktop running so I can have Perforce access when I'm away from home; it feels like such a waste. The idea of hosting my Perforce server on a dedicated server box really makes my nerd side happy :). In addition to that though, my boss pointed out I could run backups through that server as well, which is another big priority for me since I don't have any backup systems running for my home network.
I've been looking at the HP EX495 MediaSmart Home Server. It has some pretty good spec's on it, and the price is pretty good. Guess we'll see what happens
One of the biggest reasons I want a dedicated server box is so I can move my Perforce server off my desktop. For some reason I always hate having to leave my desktop running so I can have Perforce access when I'm away from home; it feels like such a waste. The idea of hosting my Perforce server on a dedicated server box really makes my nerd side happy :). In addition to that though, my boss pointed out I could run backups through that server as well, which is another big priority for me since I don't have any backup systems running for my home network.
I've been looking at the HP EX495 MediaSmart Home Server. It has some pretty good spec's on it, and the price is pretty good. Guess we'll see what happens
Saturday, May 22, 2010
Update on things
Been pretty busy lately. Had to travel to a conference for work in the beginning of May, been bouncing around different projects at work, and my buddy and I have been putting in a lot of time on our new game idea.
Things at work have been especially interesting the past week when I got moved over back fill on a project for my boss. We have a piece of software we're upgrading and preparing to license; however while the functionality the software provides is pretty good, the underlying code is pretty messy and old. The past week I've been getting oriented in this new code-base, and doing a lot of clean-ups and improvements. The MS C++ compiler (VS 2008) has been pretty handy in this regard, it spits out a lot of good info regarding deprecated or dangerous functionality. Working on this has been an interesting diversion and challenge, so I'm pretty happy doing it.
Things at work have been especially interesting the past week when I got moved over back fill on a project for my boss. We have a piece of software we're upgrading and preparing to license; however while the functionality the software provides is pretty good, the underlying code is pretty messy and old. The past week I've been getting oriented in this new code-base, and doing a lot of clean-ups and improvements. The MS C++ compiler (VS 2008) has been pretty handy in this regard, it spits out a lot of good info regarding deprecated or dangerous functionality. Working on this has been an interesting diversion and challenge, so I'm pretty happy doing it.
Saturday, April 3, 2010
iPad release day today
Apple's new iPad comes out today, that's pretty cool. I've been debating the pros and cons of getting one myself; and for now I'm leaning to not getting one for the immediate future. I'd love to have an iPad to play with and to develop games on, however right now those 2 reasons aren't quite enough to justify dropping $5-600. I'm still working on an iPhone project, and the iPad would easily distract me because I'd want to play with it.
Speaking of which, Unity3D just came out with an update to their iPhone SDK, Unity iPhone 1.7. I'm pretty stoked about that update, especially being able to play your game in the iPad simulator. That'll let developers without an iPad get a feeling for how their game will look running on the iPad device (not a perfect example though). I already downloaded and installed the update, however now I'm waiting to finish downloading the most recent iPhone/iPad SDK from Apple. Once that's finished and installed, I'm going to rebuild my current test project and upload it to my phone to check for problems. The Unity folks do a fantastic job with their updates, but every now and then a problem slips in, so it's always good practice to do some regression tests.
Speaking of which, Unity3D just came out with an update to their iPhone SDK, Unity iPhone 1.7. I'm pretty stoked about that update, especially being able to play your game in the iPad simulator. That'll let developers without an iPad get a feeling for how their game will look running on the iPad device (not a perfect example though). I already downloaded and installed the update, however now I'm waiting to finish downloading the most recent iPhone/iPad SDK from Apple. Once that's finished and installed, I'm going to rebuild my current test project and upload it to my phone to check for problems. The Unity folks do a fantastic job with their updates, but every now and then a problem slips in, so it's always good practice to do some regression tests.
Subscribe to:
Posts (Atom)