How to be a good tester? (Full Version)

All Forums >> [Current Games From Matrix.] >> [World War II] >> Steel Panthers World At War & Mega Campaigns



Message

Resisti -> How to be a good tester? (7/2/2001 3:30:00 PM)

A lot of threads on how to be a good scenario designer (thanks WB,btw!),but none on this matter. I know maybe this is more subjective than design methods, but nevertheless I'd be HIGHLY interested in hearing from some of the top (and not only) designers which kind of feedback they expect,when sendng out a scenario for testing. Maybe a sort of "rules list",altough necessarily a bit generic and subject to personal interpretations,would be useful; well it would be for me at least :) Cmon WB,Brent,Redleg,Marauder,Paul V.,and all the other design gurus, share yr toughts with us. Naturally, opinions from top testers(Warrior? Alexandra?,who else?) as well as anybody else are welcome!



Warrior -> (7/2/2001 6:44:00 PM)

First and foremost, I look for technical glitches: hidden buildings, vehicle "traps" not noted on the map, unattainable victory hexes (perhaps set on a cliff or some such where it's impossible to get to them), unit anomalies (an ammo dump with ATG's, weird weapons, etc.), and so on. I also look at the text for spelling and grammar errors. Then I try to determine if the scenario is historically accurate. I'm not as "up to speed" on this as I could be, so I generally trust the designer. I get a "feel" for the play - is it too easy, too hard, does it leave me with a feeling of identification with the units involved, etc. Did I like the scenario, did it make me crazy, would it be possible to win, was it crap? If the designer has asked for my opinion, I give it, otherwise I stick to the technical stuff. I give a brief AAR so the designer will know how I approached the tactical problems presented by the scenario, i.e., did I attack from the north or south, did I split my forces, was maneuver a big part of it, etc. There are other "intangible" factors, of course, but this is generally what I report. If a designer has follow-up questions or needs clarification, I address those. Oh, and if it's one of Sapper Don's scenarios... I naturally tell him it sucked no matter what. :D :D [ July 02, 2001: Message edited by: Warrior ]



Resisti -> (7/2/2001 11:36:00 PM)

BUMP Hmmm,it seems all designers disappeared.... :(



Larry Holt -> (7/2/2001 11:44:00 PM)

I do a lot of software testing, here is what I consider important: Be able to report real findings, that is what happened, not what you remember happening. For this, turn on logging (alt-L) and keep a paper or electronic journal. You can alt-tab the game into the background to give yourself time to take notes. Have a plan, what are you going to be looking for, balance, movement pathing, etc.? That way you don't waste time, you finish the scenario then you wonder about something and have to replay it. The designer should give you a list of changes if this is itterative testing. Verify them.



Mark Ezra -> (7/3/2001 1:46:00 AM)

Being a good Tester? Objectivity is important. If one gets caught up in the game objectivity suffers. If one likes a particular designers work (or not) it becomes very important to remain impartial. As a tester I look first at the game description. Does it give time date length and weather. Does it describe events clearly. Does it egage the reader? Finally is the lay out, punctuation and spelling (my big challange) OK. Next I review the map. Does the map use the entire surface well. Is it functional as well as interesting in both the over view and detailed view. Are roads free of hidden "stone Bldg?) are there other defects common to the map editor (like a #3 hill hex left over in your city or wheat field). Finallly can the AI move reasonable well or is it hindered by too much rough or soggy terrain. If it is a historical scen, does the map resemble the actual battlefield. Now go to the OOBs Are the OOB's for each side reasonable to the scen. If it's historical does it match known OOB's. If it's not does it have a intersting blend of units. And now we test the game!...Does it play through the lengtgh of the game? was it too easy, too hard, to silly? Did it create intensity, excitement or was it just an exercise. Did the AI act as the designer wished? If it's historical does it give the flavor of the actual event?And finally the score. How'd I do, It's always good to enclose a screenshot of the score. It will help the designer to understand the Vhex cost, casualties, units activity and is a clear proof that the entire game was played. If for some reason the game was not completed the designer needs to know. Now all that needs to be done is report the findings and recomendations in a clear logical step by step manner.



Warrior -> (7/3/2001 1:54:00 AM)

quote:

Originally posted by Mark Ezra: Objectivity is important. If one gets caught up in the game objectivity suffers... Now all that needs to be done is report the findings and recomendations in a clear logical step by step manner.
Objective? Clear? Logical?? My highest praise for a scenario goes to the one that makes me scream profanities at the monitor and pound on the keyboard & mouse! :D ;) :D [ July 02, 2001: Message edited by: Warrior ]



Mark Ezra -> (7/3/2001 2:44:00 AM)

Warrior: That's cause you're a GOOD player...LOL I'm lucky to get a draw!



Grenadier -> (7/3/2001 2:48:00 AM)

Most of the time I just want to know if the scenarios are being played :D Seriously, I look for playability of the scenario. Is it predictable? Is it accurate? The ones I like the least are the "go for the first VO's and defend until the AI is bled dry then take the rmaining VO's". The best cure for this are the time per turn VO. The feedback that helps me the most is finding out if the scenario played out historically or was the player able to change an historic outcome as most of my scenarios are designed to be recreations of actual battles, such as Seelowe and Villers-Bocage. I want to know if the VO's worked as designed, if the scenario was fun to play, if it was realistic, was the balance of forces OK? Were there any surprises? Any kind of feedback is welcomed by me. That way I know they are being played.



Don Doom -> (7/3/2001 3:02:00 AM)

I agree with DGPT[Brent], I generally look to see if the tester found it playable, if the v-hexes where reachable in the time allowed, if the map was historicly correct as possible. If the .txt file gave enough info for the battle without giving it away. The two forces are balenced. Don ps Warrior who is sapper don?



Warrior -> (7/3/2001 4:02:00 AM)

quote:

Originally posted by Don Doom: ps Warrior who is sapper don?
Don "Sapper" Llewellyn, coordinator of the SPWaW Tactical Training Center and all-round good guy.



Warrior -> (7/3/2001 4:06:00 AM)

quote:

Originally posted by Mark Ezra: Warrior: That's cause you're a GOOD player...LOL I'm lucky to get a draw!
Mark, don't worry, you're a good player. The evil, nasty designers are the ones to blame for all the draws - it's not your fault. They get a perverted, degenerate satisfaction out of making scenarios that are difficult in the extreme. :D :D :D



Don -> (7/3/2001 4:24:00 AM)

Ah, good old Warrior! We worked together so much on the MC that sometimes while positioning extra Matida's for a flank attack I'd be thinking of Steve's reaction! And wishing I could be a "fly on the wall" when they came in! :) The worst kind of testing result are the ones that simply say "I like it" or "I don't like it". These pretty much tell you nothing, so the review has to be broken down into different areas such as map, objective placement, etc. WB was the first one I saw to ask for specific things from testers, and that carried over into the MC. This way you get more consistent reports, and feedback that you can really use to make the scenario better.



Warrior -> (7/3/2001 7:20:00 AM)

quote:

Originally posted by Don: Ah, good old Warrior! We worked together so much on the MC that sometimes while positioning extra Matida's for a flank attack I'd be thinking of Steve's reaction! And wishing I could be a "fly on the wall" when they came in! :)
I REMEMBER THOSE MATILDA'S! They made me gnash my teeth, and if I had any hair I would have torn it out! :D



mr172 -> (7/3/2001 2:10:00 PM)

Ciao: come diventare un buon tester? Well (in English for all the bunch). The first step is working like a proofreader. Double check all, the mistake could be hidden also in the best scenario and sometimes the designer himself could'nt know that the bug is here. Hidden building, stream under paths, units without ammo evrything. Then play against not with the scenario. Search for weakness. There is an impassable route to get the flag? Try it, may be it was impassable only in the whisful thinking of designers. As tester you have two duties, demonstrate that the game is well balanced and in the same time demonstrate that you could anyway outbalance the game. Testers not designers are the worst friends of gamers. Poi se vuoi ne parliamo pił a lungo in privato. indirizzo mr172@hotmail.com Massimo



RockinHarry -> (7/3/2001 11:05:00 PM)

..and finally don“t forget to send the "save Score" file. :D ...at least for me there“s valuable information to be found. __________ Harry



Wild Bill -> (7/4/2001 2:54:00 AM)

Well, my turn. Resisti, you have heard from the best of the best. The testers/designers who have responded to you here are the top of the line in my opinion. Nearly all of them have tested for me and any success I enjoy in my efforts I attribute to this group of "unsung heroes." Testing can be tedious. It can be frustrating. That is because you are dealing with an unfinished product. But it can be also a lot of fun...a first peek at what is coming. It also gives a real sense of satisfaction in knowing you have contributed to the gaming community, often in an anonymous way. That is why I say that testers are selfless heroes. If the light ever shines on me, I think it should shine even brighter on these great gamers who give time and effort to make your wargaming experience so much fun! I salute you all! Wild Bill



Wild Bill -> (7/4/2001 3:09:00 AM)

Having said all that, let me pass to what I expect from those who work with me in testing scenarios. 1. I need durability. Too many volunteer...then disappear! :( Count the cost before you begin. 2. I need communication. There are testers who are listening but never responding. I can't possibly know what is going on unless you write me. This is perhaps my second biggest peeve: testers who don't communicate :mad: 3. I need thoroughness. Check it all. Testing is much more than just playing the battle. It requires a sharp eye, a look at detail and the mechanics of the battle/campaign. As has been stated, you start with the text. Read it. Check it first for grammar mistakes. note them. Read it again to get the feel of the battle. Does it give you that? Is there too much? Is there too little? Does it sound accurate? Now open the scenario. You've read the text. Now play the battle. Play it through. Note your score. Note anything wrong you see as you play. Make notes. If not, you'll forget. I know this is different from what most do, but this is the best way. Now that you have played it and gotten a your first impressions of it, open it in the scenario editor. The first step is a hard look at the map. Any roads incomplete? Are there hidden building hexes? Does it have the look and feel of real terrain? Does it fit the situation? Next, look at the smaller operational map to the right? Little dots indicate troop locations. Do you see any that are still along the edge of the map? Should they be there, or did the designer miss them when deploying units? After playing the scenario and looking it over, and IF you have the time, play it again. Make more notes on what you find. Keep your final score. Next, write out your report. Use common courtesy. DON'T say, "This is a piece of crap!" (even thought it might be). Be diplomatic. Find something nice to say about the scenario before you begin, as we say here, "cracking" on the work. Let your criticism be constructive. Offer your own ideas on how it might be better. Examples: It should be longer - shorter. Units should be closer or further away from the battle area. Reinforcements should come in earlier or later. Place reinforcement flags further away or closer. Modify artillery or air support. If it is unbalanced, how could the designer make it more balanced (challenging, yet winnable)? Remember, even if you make suggestions, the designer may not use them. Don't get offended. He has that right. His name is going on it. If, however, you feel you are wasting your time trying to help a designer, just don't do anymore work for him. These are a few haphazard thoughts for you Resisti to get you lined up for what you are about to do. Good luck with it. Wild Bill



Don Doom -> (7/4/2001 3:16:00 AM)

Amen. ;)



Don -> (7/4/2001 9:30:00 AM)

One more thing - testing is one of the best ways to learn scenario design. I realized that testing for WB would teach me alot, so tested a few campaigns and it really paid off in seeing how he did things. Tester's reports are also different, I think, depending on whether or not the tester is also a designer. I think it's natural that Alex would suggest improvements while Steve wouldn't, because Alex is also a designer and probably looks at the scenario a little differently. I don't see anything wrong with suggestions, myself. the designer can always not implement them. Some testers also have a problem telling the designer if they don't like something - I think this is your job! If I think the map is crappy I say so - if I'm asked to test I figure it's my job to comment. But WB is right, it has to be done with something called tact, which I have yet to locate any of. Hopefully when I do I'll get some! :D



mr172 -> (7/4/2001 1:26:00 PM)

Just a nuance over the Wild Bill statement. I never read the text before playing. I will play with "ingenuity" because text often give me an other side of the hill look into designer ideas. So I first open the battle screen to the minimum zoom. General overlook of the battlefield, where I'm where I must go, then I zoom for the "trouble", trees along a road, buildings that can be useful for ambushes, dominanting hills... Then I stop and start to think. what I'll do if I was the designer, where I put units, where and when the reinforcement could arrive... Then again what I, by sure, would'nt. This is usually what the designer has really done :D Okay now check for the mistake. Scroll the map along the possibile way of approach, use a fast delay time and put the hex grid on the map, read carefully the black boxes. The full control with the editor could be done, as Bill says, after the battle. Ciao Massimo Rocca



Warrior -> (7/4/2001 6:16:00 PM)

quote:

Originally posted by Don: ...it has to be done with something called tact, which I have yet to locate any of. Hopefully when I do I'll get some! :D
Let us know if you find any, Don, it will be a great relief to many. (And a cold day in hell! :D )



Wild Bill -> (7/4/2001 6:44:00 PM)

Yes, Sapper. Telling what I want to hear is not telling me what I need to know :D . But telling me in a constructive way is a pill I can swallow...well, might choke a little ;) . As the Book says, "The wounds of a friend are better than the kiss of an enemy." Wild Bill



Wild Bill -> (7/4/2001 6:45:00 PM)

Interesting approach, Mad Italian. My problem with that is sometimes I can't figure out what the heck I am supposed to do to win ;) But not reading the text will definitely lead to some surprises. Your method of analysis sounds good to me. The Kunel



Don Doom -> (7/4/2001 6:45:00 PM)

Watch it warrior or I will send you back to ft stalin :D :D don



skukko -> (7/4/2001 9:13:00 PM)

Finally I have time to look at here and what do I found: "How to become a good Tester ?"-topic :D At the beginning its enough that you can play it and find some obvious errors, the ones that you find while playing. This covers technical issues as well tactics of the AI (designers side). When graduated in this you'll need to have a designer living inside you to get to next phase of a testers endless run. When you graduated first class you have already enough info of how scenarios are done. Unless you have found designer who don't give you feedback. I personally try to give feedback of reports and scenarios as well from enhancements that I'll going to do and ask cold opinion from tester of them. Suddenly along the route you'll note that you ain't plain grunt whos testing but you want to design your own stuff. Then you'll enjoy company of other designers that have run out of testers. Its a loop :D I personally do as mr172 and don't read txts before I have started and decided how I'll going to play scenario. Usually afetr the game is ended I'll read txt-file and by that I know did it work as described and designed. Now its time for editor. At the editor I do look over AI Forces 1:Mixture; is it suitable for its mission. (I do give a bone to balanced scenarios :D as they are usually (balanced) meant to be played with one tactic and result is that rest of the tactics don't work as designed.) 2:Stances of the troops and reaction times. This because too many good scenarios have been tested many times because designer have forgot to check these two times before saving game. Even the best designers leave these sometimes unchecked. 3:Objectives. Does they fit in overall madness of designers wants and needs. Usually they are left to 0.0 or that old 222.222. (Ok, leaving objects open is nice way to add some suprise -elements to AIs behavior but not when they are left as is by accident.) 4:I create two hexes long 'secondary road' or what road is used at the map. This removes all roads that look like roads but are nothing. Bad guy ? :D 5:I look after map-errors. These can be design problems ( map has one small, rough, high hill in the middle of the open area and all roads are drawn over it :eek: )or then accidentally created (multibuildings and collapsing hexes) :rolleyes: And now I do same with the side that is designed for human player. Maybe I will be designer some day and enjoy to do tricky scenarios for testers... mosh



Wild Bill -> (7/4/2001 11:35:00 PM)

Jump in the water, Mosh! It ain't that bad! I'm sure you'll do some top-notch battles! Wild Bill



Don -> (7/5/2001 4:00:00 AM)

Things I look at while testing: First, I play the scenario! :) I look for obvious mis-matches in experience or moral, how the game plays and how the map looks and plays. Second, I open it up in the editor. Here I look to see that all Player 2's units are "P2". I look to see how many reinforcements were slated to appear, and remember how many actually appeared. Sometime the reaction turns are not set correctly. I'll also look at the map and see what I can see. Third, I open it up in WAWED, where I get a good look at all of the experience and moral numbers. Sometimes these are out of whack because the designer just used default numbers, so there is a huge spread with "50's" fighting next to "90's" when the intent was that they all be of similar numbers. Fourth, make a good, detailed report commenting separately on the map, the overall gameplay, and then go into any detailed "issues" I found (or think I found :)). Also include the score or scores from testing, and comments on the text file - sometimes they are not clear or contain spelling and grammer errors. If you have inexperienced testers, it is best to give them an outline of the kind of results you expect - otherwise you may not get the results you want.
quote:

Let us know if you find any, Don, it will be a great relief to many. (And a cold day in hell! )
Actually I was thinking of you when I wrote that! :D No, we've both "erred" and said things in the heat of a test that we regretted, and a valuable lesson it was! :rolleyes: Too bad email doesn't have a "tact" button to change questionable text in oh-so diplomatic phrases! "Man, this scenario sucks!" would become "You know, I really liked this, but here is what I would do...". Ahem.



Page: [1]

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
2.796875