|
Erik Rutins -> A general post on PC Games and bugs... (8/10/2009 8:19:10 PM)
|
Just felt the need to pontificate a bit today... nothing in specific brought this on, it's just been percolating for a while and some thoughts accumulating based on many posts I've read here and there. So here is an extended ramble on the subject which I hope will be illuminating to some and at least won't put too many of you to sleep or irritate you. This is basically stream of consciousness and not intended as a whine, but rather to inform as a number of the posts that made me want to write this were from customers who may not know what goes on "behind the curtain".
Developing software for the PC is a lot different in one sense from developing many other products. Your "platform" is a moving target with nearly infinite permutations. If your company makes kitchen scissors, you can reasonably assume that you will not have a customer who has hooves instead of hands or who wraps his food in sheet metal that your scissors can't cut, instead of aluminum foil.
Similarly, in a much closer example to PC games, if you develop games for the XBox, you can expect that your customers will all have the same XBox manufactured by Microsoft with the same hardware. Once the hardware changes, it's a whole new platform (i.e. XBox 360 or Playstation).
What I'm getting at is the perhaps now old yarn that developing for PCs is different. Not only is every PC virtually unique in its own combination of hardware and some of them are home built by folks with just enough knowledge to be dangerous, you have on top of that the drivers (which are in fact the cause of the majority of our technical complaints) as not only do folks not keep them up to date (many don't even know how) but even some of the up to date drivers have bugs or perform differently from the spec they are supposed to math.
Then you've got the operating system and its components, which come in a variety of flavors, with a whole new layer of complexity based on which version of Windows, which Service Pack, what Hotfixes may or may not be applied, what version other components like DirectX and .NET are at, etc. Finally, you have the third party software. Over 90% of our help desk complaints that read like "I can't get the game to install or run." are because someone has their computer's anti-virus/security software setup so that it won't even allow any programs to read or write to any part of the registry.
If we could convince the OS and Hardware manufactures to somehow automatically keep people's drivers up to date and we could convince the third party software makers to have their default settings be a bit less aggressive, I think our Help Desk volume would easily drop by half or even more.
Ok, so with all this said - right, we know that, we got in to the business knowing it and as professionals we do our best to make sure we work around it. Nevertheless, you have to keep it in mind because the reality is that we simply can't test on every single PC and we can't play every game exactly the way every single customer might play it, so things will slip through.
Of course, then there are the real bugs that we created. They exist and I don't think I've ever seen a piece of software that didn't have one, the real question is how many there are and how frequent or severe they are. Even in non-complex games, our pre-release testing often finds many, many bugs and sometimes fixing one bug creates another. We don't release a game until we are happy that as far as our own testing can determine, there are no bugs to be found. Unfortunately, this is as much an art as a science and while I'd say we are much better at it than we were when we first started doing this, there is still an aspect of it that comes down to a judgement call. We are more successful with that judgement call on some games than on others, so we've had some releases where it turned out we really had found 99.99% of the bugs pre-release and fixed them, but others where we only found say 90% and a lot needed doing post-release. So, as I said you release and then the universe re-confirms for you the difference between say 20 testers and developers and a few thousand very motivated customers is - it's big. Especially when you mix in the hardware/software/driver/OS stuff I mentioned above. And yet, even if the customers find some bugs, there's no guarantee that if we'd kept it in testing for another month or even a year, that with our mix of systems and tester styles we would have found the same bug.
This is the part where I think post-release support is simply necessary and a fact of life. We recognize that it is simply not realistic to expect that because a game seems "bug free" to us that it actually will prove to be bug free once customers start using it. We strive for that, but we try to provide the best possible support to quickly fix issues after release and make sure that if something slipped through, it's addressed quickly. I feel that this way customers know that even if they hit a bug, the game will be well-supported and that bug will not survive long or affect their enjoyment of the game long-term. This to us is the only way to truly make a game effectively "bug free" and its part of realizing that releasing a PC game "into the wild" is much like going into battle. Whatever your plan was before, whatever you thought you knew, you will have to adapt and change if you are to succeed.
On top of this, you have the "bugs that are not bugs", areas where a customer either disagrees with the design of the game or where he had different expectations. I think of these as "issues" and we try to address as many of these as we can, to make customers happy while keeping to the vision of the game.
For example, on our most recent release of WITP Admiral's Edition, we have not had a single confirmed crash issue yet. However, we had some interface lag issues come up (already partially addressed with a hotfix) due to how the game interacts with some kinds of dual core/quad core chipsets. Despite the fact that we have dual core system and even had a quad core or two on the testing team, this did not come up, as it was not an issue on those particular systems. There are some other data typo issues and balance tweaks we are working on, but overall I'm confident that we would not have found 99% of the reported issues so far by keeping it in testing any longer. The first release is an enjoyable, stable game that a lot of customers are getting a lot of entertainment value out of. Now our job is to improve the game post-release as quickly as we can.
Nevertheless, a customer could reasonably say "well look, they released another game with bugs". And that same customer could look at his XBox game (note that they do have bugs too, but generally are more bug free for the reasons noted above) and say "why can't these guys make a game without bugs like this one"?
I guess my point is that if making games for the PC were just like making games for the tabletop or for some other platforms, or making some other products, then we wouldn't have to deal with bugs either, or at least far fewer bugs. But the nature of PC game development and especially niche "independent" development/publishing where the development and testing teams are by necessity smaller due to smaller budgets, is such that it seems there will always be some bugs, even in a well tested, well-programmed game.
Sitting where I sit, I also know how many issues we do catch and address before release and I know that we never "pull the trigger" on a release until we're as sure as we can be that it's really ready for the customer to dive into. I can't promise our customers that every game will be bug free, I truly think that's impossible and I also know that we have made some mistakes in the past where a game was not tested thoroughly enough and we made a release decision based on incomplete information (we have learned from those mistakes, fortunately). I can say that we will do our homework before release to allow as few bugs as possible to slip through and we'll work with each developer post-release to provide the best possible support so that any issues are fixed as quickly as possible and your long-term enjoyment of the release is assured.
I will also say that, as many of you also do, I buy "mainstream" games. Some of them just interest me, others I buy to keep up with the "state of the art" and to see what the AAA publishers and developers are doing. Over the last few years, with a few exceptions, every such release that I've bought has in my experience been more buggy at release than our games are and often less well-supported. I have sympathy for the AAA publishers and developers, in the sense that they are pushing the cutting edge hardware/software wise much more than we are and thus are dealing with many more situations where drivers or other components just can't keep up. They also have to deal with retail deadlines and unfortunately retail in this day is a very unforgiving environment for PC games and the retail deadlines and policies probably cause more "buggy" releases than anything else. This is one of the reasons why we send fairly few games into retail these days.
Anyway, my point on that last general comparison is just that I think we stack up pretty well as far as delivering quality games at release and supporting them well, but we will continue to try to improve and find ways to be better at what we do with the resources we have.
If you read this far, thanks for listening.
Regards,
- Erik
|
|
|
|
