coregames
Posts: 470
Joined: 8/12/2004
Status: offline
|
quote:
ORIGINAL: WIF_Killzone
Forgive me if I am naval gazing, my mind is racing, not a good thing.
The question i think I am asking is how to develop a list of performance indicators or measurement criteria so that Matrix has the right type of information to gage the quality of the AI. i.e. how to determine if its ready for release. This could be be done via a percentage of test cases successful (e.g. We will release the product if 98% of strategic test cases function as designed, etc)
How then will the test cases be developed to test the AI?. In normal software development there would be "UAT" (user acceptance testing) that follows system testing (developer and test team testing). A series of test scenarios comprised of a number of test cases would be created.
An example scenario may be "AI playing Germany attempts to repel invasion of France". A series of pre-conditions would be specified (i.e. France is under german control, Vichy France has been established, germany is at war with Russia, etc, as well as expected results (i.e., Germany does abc, xyz, etc...attempts to repel invasion...deosn't sacrifice russia war to0 much?.. these would all be quite "strategically"broad based scenarios. The test cases then would be more detailed, as well as the pre-conditions, and 'expected results'. Quite a lot of work as you can see. Not to frighten, but I have seen testing budgets as large and larger then requirements, design, and development budgets (for example in banking or insurance where a mistake could prove fatal or cost the company a lot of money) I digress.
Conversely, would test cases simply be created to test and verify the AI's strategic, tactical and operational level "thinking" is working as designed? What would these look like?
I am curious on what the approach will be for testing the AI. Should test planning, scenarios and test case development be started now (if that is the approach that is used), and how could the forum support this work, will additional beta testers be required, has the approach been given any thought yet?
Hmmmmm.
If Steve does adopt some form of rubric for 'grading' the performance of the AIO in various situations, in my opinion the grade should be based on a personal assessment rather than some sort of strict formalized criteria. The game is just too big to lend itself to easy quantification, and players (especially experienced WiFers) will have the ability to think at a higher level in order to judge the AIO's performance, which may not be reflected in the outcome of a scenario, depending on luck among other factors.
_____________________________
"The creative combination lays bare the presumption of a lie." -- Lasker
Keith Henderson
|
Printable Version
AI Acceptance Criteria - 
New Messages
No New Messages
Hot Topic w/ New Messages
Hot Topic w/o New Messages
Locked w/ New Messages
Locked w/o New Messages
Post New Thread