RE: Playtesters?

Matrix Games Forums

Forums Register Login Photo Gallery Member List Search Calendars FAQ

My Profile Inbox Address Book My Subscription My Forums Log Out

RE: Playtesters?

View related threads: (in this forum | in all forums)

Logged in as: Guest

Users viewing this topic: none

Printable Version

All Forums >> [Current Games From Matrix.] >> [World War II] >> Carriers At War >> RE: Playtesters?

Page: << < prev 1 [2]

Message

<< Older Topic Newer Topic >>

RE: Playtesters? - 7/3/2007 6:27:24 AM

MarkShot

Posts: 7089
Joined: 3/29/2003
Status: offline

Very funny!

The stories I can tell about the gaming industry is nothing compared to those I have about trading floors, pharmceutical companies, commercial banks, big accounting firms, software consultancies, Internet startups, VC funding, dot-coms, ... If you were to hear some of those stories, then you would know that the PC game industry represents the very highest ideals of integrity and delivering value to the customer; at least, in relative sense. You got to be big to steal big. The gaming industry is too small potatoes to take seriously.

_____________________________

(於 11/13/21 台北，台灣，中國退休)

(in reply to Prince of Eckm�hl)

Post #: 31

RE: Playtesters? - 7/3/2007 6:31:35 AM

MarkShot

Posts: 7089
Joined: 3/29/2003
Status: offline

By the way, I agree that we should all get back to playing CAW or whatever. This thread has gotten much too serious for a group of middle aged men clicking on little toy ships on a map of the Pacific! :)

_____________________________

(於 11/13/21 台北，台灣，中國退休)

(in reply to MarkShot)

Post #: 32

RE: Playtesters? - 7/3/2007 6:34:19 AM

Prince of Eckm�hl

Posts: 2459
Joined: 6/25/2006
From: Texas
Status: offline

quote:

ORIGINAL: MarkShot

By the way, I agree that we should all get back to playing CAW or whatever. This thread has gotten much too serious for a group of middle aged men clicking on little toy ships on a map of the Pacific! :)

Agreed!!!

PoE (aka ivanmoe)

_____________________________

Government is the opiate of the masses.

(in reply to MarkShot)

Post #: 33

RE: Playtesters? - 7/4/2007 3:30:42 AM

JD Walter

Posts: 235
Joined: 6/20/2003
From: Out of the Silent Planet
Status: offline

* ...Ahem...*

If I may beg the Gentlemen's indulgence (before returning to our regularly scheduled game of CAW), I would like to add the following in this regard...

quote:

As long as I am handing out blame, then the gaming community review web sites are to blame too. Very few of them will really expose a game as unplayable pile of CTDs and bugs. Very few of them will bury a product for being a cinch to beat after only a few days worth of play experience. So many of these sites are whores who are in bed with the products they are responsible for objectively reviewing. They are not consumer advocates, but in fact just extensions of the marketting apparatus of the industry. Look at some games which are a few years old that are widely known to be only mediocre. Now, go back and look at the reviews which they games received a few years back. For the vast majority, the review will relect a much more upbeat assesment than the product actually deserved.

I have a friend who presently works in the game review magazine industry. She recently changed publications. After this move, she relayed to me that, at her previous publication, due to the pressing need to be the first to print with a "timely" feature on whatever was released that month, the majority of reviews were based on a single playing of the game at the easiest possible setting. This was generally completed in one day, then used as the basis for a 250-300 word "review".

Only the most egregious faults were ever seen, much less written down and reported on. She remarked that in her own personal experience, only a CTD or BSOD would warrant a note that the game had a "bug".

After this, it was thrown on the shelf (or, more accurately, into a pile on the floor of a spare office), and it was back to WOW for the rest of the workweek.

Employees who were given two such writing assignments were considered overworked. They would complain loudly about losing 16 man-hours levelling up their character(s) in WOW, and falling behind the rest of the office.

This trend got so bad, some reviewers started quoting from internet sites instead of playing the game thoroughly themselves, in order to catch things they might have missed in their cursory look at it for the article, so that they could be sure to note something in their write-up that was common knowledge on the 'Net, and not be embarrassed at missing an obvious flaw that was becoming too well-known and talked about in the "early adopter" gaming community.

Moral: Readers who subscribe to such magazines should be aware that many print reviews are based on, at most, an eight-hour session with the game, at the easiest setting allowing the ending to be played. Be forewarned.

We may now return to our appointed forum ...

(in reply to Prince of Eckm�hl)

Post #: 34

RE: Playtesters? - 7/4/2007 4:32:04 AM

MarkShot

Posts: 7089
Joined: 3/29/2003
Status: offline

Time to go back to playing. Look at this disaster which has just befallen me at Wake! :(

Attachment (1)

_____________________________

(於 11/13/21 台北，台灣，中國退休)

(in reply to JD Walter)

Post #: 35

RE: Playtesters? - 7/4/2007 5:03:19 AM

Fred98

Posts: 4430
Joined: 1/5/2001
From: Wollondilly, Sydney
Status: offline

I was not a tester on this game.

But, the other thing about testing, it �s usually impossible to complete a scenario H2H because the developers release the next beta version before you get too many turns in.

As a result only the first 6 or 8 or 10 or so turns of any scenario get tested H2H so its difficult to check the balance of longer scenarios.

-

_____________________________

(in reply to Prince of Eckm�hl)

Post #: 36

RE: Playtesters? - 7/4/2007 5:23:20 AM

Oleg Mastruko

Posts: 4921
Joined: 10/21/2000
Status: offline

quote:

ORIGINAL: Joe 98

But, the other thing about testing, it �s usually impossible to complete a scenario H2H because the developers release the next beta version before you get too many turns in.

As a result only the first 6 or 8 or 10 or so turns of any scenario get tested H2H so its difficult to check the balance of longer scenarios.

Very very true, but not for this game

_____________________________

(in reply to Fred98)

Post #: 37

RE: Playtesters? - 7/4/2007 5:25:44 AM

Oleg Mastruko

Posts: 4921
Joined: 10/21/2000
Status: offline

quote:

ORIGINAL: MarkShot

Time to go back to playing. Look at this disaster which has just befallen me at Wake! :(

Don't know about beta testers (after all it's the thread about them

) but developers don't think it's a disaster at all! Of course I am talking about CAW 2007, not CCAW. For more details see my thread "Ridicolous scenario design" (should be on a page 2 or 3 by now).

_____________________________

(in reply to MarkShot)

Post #: 38

RE: Playtesters? - 7/4/2007 5:43:05 AM

Duck Doc

Posts: 693
Joined: 6/9/2004
Status: offline

You are kidding me, right?

Let's see: we have found the one area of human concern magically sequestered from the foibles of human nature

. Everybody checks their character defects at the door when beta testing - or developing games. No brown nosing? No fragile egos? No favoritism? Come on. Who are you trying to kid? (I'm guilty even & I ddn't even test in the real sense.)

This is the real world. All is corruption. From the top of the govenment down to the guy who mows your lawn. Your senator & congress person, your hairdresser.

Besides the testers are confined to the universe the developers create. There are constraints on criticism. Check out the difference between how the ciriticism is recieved on the various newly released games on the forums here.

Now, nobody is making a killing making wargames we old geezers like to play. Let's not make a federal case out of these things. Let's lighten up a bit. Let's cut everybody a bit of slack: the developers, the beta testers & the players. Let's have some fun...

...but let's not go on applauding the emperor's new clothes.

My problem is I can't keep my mouth shut. I never did & never will. Too much integrity for that. I'll never learn. Hard to get ahead this way...

.. but I digress

quote:

ORIGINAL: MarkShot

Very funny!

... then you would know that the PC game industry represents the very highest ideals of integrity and delivering value to the customer; at least, in relative sense. You got to be big to steal big. The gaming industry is too small potatoes to take seriously.

(in reply to MarkShot)

Post #: 39

RE: Playtesters? - 7/4/2007 6:41:22 AM

Prince of Eckm�hl

Posts: 2459
Joined: 6/25/2006
From: Texas
Status: offline

quote:

ORIGINAL: Joe 98

I was not a tester on this game.

But, the other thing about testing, it �s usually impossible to complete a scenario H2H because the developers release the next beta version before you get too many turns in.

As a result only the first 6 or 8 or 10 or so turns of any scenario get tested H2H so its difficult to check the balance of longer scenarios.

-

There was a game that I tested that had a scenario that was three to five hours in length depending on the variant that one might draw. The maximum time compression was 3-1, so I was looking at least an hour of testing to reach the conclusion of the game. Toward the end of play-test cycle, a bit of a crisis arose because the testers were reporting the game ending without the scenario results being displayed.

Anyway, While everyone and his uncle reported to their buddies for a Sunday's afternoon of NFL football, I tracked that bugger down and uploaded a saved game clocked to three seconds before the failure, and the issue was quickly resolved. That's playtesting.

Even using Run5, I could have done the same with CaW-007 in a TINY fraction of the time that I had devoted to the bug in the other game, the key being the ability to accelerate the game clock. If I've been short with Gregor at times, that's why. He or his crew ought to be able to reproduce these bugs, more or less effortlessly, with a minimum of time and effort expended. That their customers are asked to do it, I believe, is unacceptable.

PoE (aka ivanmoe)

_____________________________

Government is the opiate of the masses.

(in reply to Fred98)

Post #: 40

RE: Playtesters? - 7/4/2007 7:27:02 AM

MarkShot

Posts: 7089
Joined: 3/29/2003
Status: offline

My profession isn't actually computers in the sense of programming. Of course, I did some of that starting out many years ago. But what I really do is design systems.

One of the basics of good system design is the assumption that the system will have both execution defects (things causing crashing and other undesirable actions) and logic defects when the system just fails to function as specified. Most programming environments provide various tools for debugging programs. However, a system usually should be designed with built-in testing tools to help the developers track down problems. These test tools will be more oriented towards the purpose and functionality of the system than simply displaying the values of variables and whether a particular code path is taken.

A game is nothing more than a relatively simple system architecture. As complex as customers think they are, they tend to be much simpler than a trading floor or most commerical systems. In any case, the game should include facilities for logging critical data to simplify the problem of tracking down bugs and behavioral problems. The game should have facilities to exactly replicate the same sequences of events over and over again (and remove random behavior; good for playing, but bad for debugging). The game should have facilities to capture key stroke/mouse logs so that user inputs can be replayed to the engine. Logging and data capture tools (with replay capability) become even more important for multi-player testing, since the concurrency adds all types of timing related problems such that traditional debugging would completely screw up the sequencing of events due to the long delays it imposes during to programmer interactions.

The bottom line: A good design anticipates problems and builds tools into the design for reproducing, investigating, and resolving problems. Poor designs rely on nothing more than traditional debugging tools and brute force. In the end, any money saved by reducing the amount of development effort coding custom debugging tools is lost during the increased time needed for defect resolution.

_____________________________

(於 11/13/21 台北，台灣，中國退休)

(in reply to Prince of Eckm�hl)

Post #: 41

RE: Playtesters? - 7/5/2007 9:00:44 AM

GoodGuy

Posts: 1506
Joined: 5/17/2006
From: Cologne, Germany
Status: offline

quote:

ORIGINAL: MarkShot

The game should have facilities to exactly replicate the same sequences of events over and over again (and remove random behavior; good for playing, but bad for debugging). The game should have facilities to capture key stroke/mouse logs so that user inputs can be replayed to the engine.......[].

Right, that's how it should be. That makes me curious though, the AA engine doesn't have this, right (i.e. built-in crash-log feature containing key/mouse logs)?

quote:

The bottom line: A good design anticipates problems and builds tools into the design for reproducing, investigating, and resolving problems. Poor designs rely on nothing more than traditional debugging tools and brute force. In the end, any money saved by reducing the amount of development effort coding custom debugging tools is lost during the increased time needed for defect resolution.

The question is if programmers have the time / ressources / knowledge to incorporate sophisticated tools (maybe even with additional AIvsAI checks) and if they use/like this method/approach. My guess is that many small devs are using basic/traditional debugging tools, for various reasons.

< Message edited by GoodGuy -- 7/5/2007 9:12:32 AM >

_____________________________

"Aw Nuts"
General Anthony McAuliffe
December 22nd, 1944
Bastogne
---
"I've always felt that the AA (Alied Assault engine) had the potential to be [....] big."
Tim Stone
8th of August, 2006

(in reply to MarkShot)

Post #: 42

RE: Playtesters? - 7/5/2007 9:34:26 AM

MarkShot

Posts: 7089
Joined: 3/29/2003
Status: offline

AA Engine:

The debugging builds do produce UI interaction logs which can be used to reproduce problems spotted by beta testers with all randomness (the seed is fixed) removed. However, the dreaded "out of sync" bug has at time compromised this capability. This code is removed in production builds (gold release and patches), since it adds a performance overhead that the customers would not really appreciate. Additionally, the beta builds contain a large collection of sanity check assert statements to catch conditions which would lead to CTDs. Once again, this code is removed for production, and if corrupt data appears, then the customer just simply sees a CTD.

I must say that Paul Scobell who did most of the programming on the engine was an extremely talented and meticulous programmer. While he was working full-time for PG (RDOA and HTTR), the quality of the beta builds often exceed the quality of many commercial production products. He was from the old school of programming; the same one I was raised in. To have anyone else catch a bug in your work is a personal disgrace. You don't need an SQA department, since it is your responsability to make sure it is defect free.

My guess is that most don't:

The primary reasons being a lack of experience in designing systems "in the large". There is a big difference between simply writing programs and designing entire systems for testability. Such systems design experience is extremely valuable for games development, since like commercial systems, most games continue to evolve the same code base year after year. So, the quality of structuring that code base for long term development often determines how feasible it is to bring out each successive game.

AGE Studio is a good example of a very well structured code base. AGE Studio unlike many developers is able to back port most new features being added to the engine for A-ACW to the original game, BoA. This is the result of a well thought out modular design.

< Message edited by MarkShot -- 7/5/2007 9:39:46 AM >

_____________________________

(於 11/13/21 台北，台灣，中國退休)

(in reply to GoodGuy)

Post #: 43

RE: Playtesters? - 7/5/2007 11:13:08 AM

GoodGuy

Posts: 1506
Joined: 5/17/2006
From: Cologne, Germany
Status: offline

I hate to quote myself, but I have to specify my question:

quote:

ORIGINAL: GoodGuy

the AA engine doesn't have this, right (i.e. built-in crash-log feature containing key/mouse logs)?

I meant a crash-report capability that could provide online feedback from users/with the user's logs. I know that nothing like that comes up, but wondered if it was part of an unemployed engine feature, besides an offline-version (logs) being employed in the beta builds.

quote:

ORIGINAL: MarkShot

My guess is that most don't:

You mean most don't use debugging tools?
In my experience, quite a few niche-devs (wargame-niche isn't the only niche) use debugging features to some extent, be it tools accessible to devs only, or tools accessible to testers. Some licensed engines even come with a myriad of debugging features and major QA divisions have customized testing routines.
Like I said before, incorporation of sophisticated tools and sophisticated software design (like you said, designing things "in the large") may not be found in various small devs's projects, but may not be found at major devs using engines off the rack, too. Some major devs have to create a "stripped" version after obtaining a license (an engine), so it's like they have to "gut" most of the core code in order to achieve certain customized effects/results, but they can still use the tools.... so it's like many of these engines "support" some level of lazyness. Some of the most prominent examples of such customizations (plagued with bugs and sloppy work) were SWAT 4 and Tribes 3, highly customized, but based on a stripped UT engine.

So you may be right when you're saying that "old school" programmers might have the better approach.

Back to wargaming... that said, I wonder if the devs of COI, for example, will be able to come up with a completely new engine/approach or "just" with a revamped one.

< Message edited by GoodGuy -- 7/5/2007 12:29:54 PM >

_____________________________

"Aw Nuts"
General Anthony McAuliffe
December 22nd, 1944
Bastogne
---
"I've always felt that the AA (Alied Assault engine) had the potential to be [....] big."
Tim Stone
8th of August, 2006

(in reply to MarkShot)

Post #: 44