|
jchastain -> RE: Building the MWiF Test Plan (9/6/2007 5:16:19 PM)
|
Grayshaft,
I think we are essentially in agreement, but let me touch on a few of your points.
First, I agree that you don't want to target boiling the ocean. My intended point was not that you have to meticulously script everything, but rather that you will be more effective if you define your scope of testing prior to beginning your planning. Perhaps the rules is the only thing you want to script out. That's fine - that's a valid decision. But I was suggesting that you consider all of the testing needs and ensure you are making that as a decision instead of just ending up there from failure to consider the broader picture. The other items will still need some type of testing and you'll want some type of feedback mechanism to encourage people to be looking at those things at some point in the cycle.
With regard to a "Testers 101" course, part of the reason I chose to detail out the basics of testing in a rather pronounced manner was to allow less experienced testers to follow the discussion as I hoped it might provide them with some additional perspective. As for access to the software, I believe you and I were both referring to the executables rather than the raw code. I agree that it would be crazy to make the code available to volunteer testers (except for snippets posted in response to specific questions). I continue to believe that the test planners should experience the software, its mechanics, and its UI. But as I have indicated that is just a part of the rationale behind the recommendation. Let's face it, volunteers want to be on the "inside" and WANT to experience the game. That's part of the "payoff" to many if not most. In the real world, test planners make more money and being a test planner represents career progression. But in terms of game testing, you are asking people to do something that is more work without offering them the "reward" afforded the "lesser" position of tester. Even if you said that this was a pathway to becoming a tester, I could see the deferred compensation model working. But to say this is something different that requires more work, does not provide the perceived benefit, and never leads to that benefit, just doesn't seem to provide the proper motivational incentives in my mind. I understand why you don't want people "promoted" to being testers as you don't want turnover in these positions, but I really believe that without any payoff you'll end up with more turnover rather than less. And I understand why you do not want to make the planners testers as well for fear they will ignore the planning and be drawn more and more into the "fun" of testing. But I continue to believe that managing that dynamic is easier than managing retention in a role that never gets to the "fun" part. Again, that's just my opinion on the matter and I'll drop it now (as continuing this discussion in public does not aid your cause).
Of interest, you also asked the question of what other games have attempted such a structured testing approach. Over the past 15 years I have been a volunteer tester for more than a dozen projects and I have never seen any of them even attempt this type of professional organized approach. As you suggest, you cannot turn it into "work" so that the testers feel as if they are operating within a straight jacket and not having any fun. But I would expect you to get some real benefit out of this approach if you can get it properly organized and then execute it in a properly relaxed manner. When you think about it, it is not entirely dissimilar to the write-up activities where people are assigned units of work and then perform them at their leisure. And that is why I continue to believe that Mziln had pretty solid insight in his recommendations to Graf regarding eventual beta participation for those who "prove" themselves through the write-ups effort.
Finally, I am afraid you may have misunderstood my points with regards to UI testing. Though I continue to believe that UI testing is important, I was not suggesting that you have a full blown UI lab. Instead, I believe your first opportunity is new testers. Whether now to WIF or not, that will be new to the UI and their initial impressions should be captured. I was suggesting you give them a list of 10 or 20 basic functions to try and perform without reading any manual or notes. Tell them to take no more than 60 seconds attempting each action and then report back that doing so was:
- 1 - Very Easy. I was able to perform this action on my first attempt with little forethought.
- 2 - Relatively Easy. I was able to perform this action fairly quickly but had to think about it a bit.
- 3 - Somewhat difficult. I was able to perform this action but it required several tries and/or UI exploration.
- 4 - I was not able to complete this action within 60 seconds.
While valuable, self reported statistics such as these have 2 drawbacks. First, we humans have ego and often want to ensure we aren't seen as the dumb one who couldn't figure something out. This will sometimes cause you to get a picture that might be more rosy than the reality. But also, you don't get to see what people actually try. If everyone puts down Relatively Easy for a given function, you might think you are in pretty good shape. But by being to watch them, you might find they all tried the same thing the first time, in which case you may want to change the UI to make the software work in the way people are intuitively attempting to use it. You only get that information when you can watch their progress against the tasks. Obviously you do not have the ability to rent out a full-blown UI lab. That is why I suggested friends and family. If you know people locally who play games but are not yet accustomed to this title, be they friends, family, neighbors, unaffiliated matrix people at an event, forum members who happen to live nearby, whatever, then you may want to have them come by and just watch over their shoulder as they attempt to do the actions. It isn't a lab, but it is free and might provide some useful insights.
|
|
|
|
