Matrix Games Forums

Forums  Register  Login  Photo Gallery  Member List  Search  Calendars  FAQ 

My Profile  Inbox  Address Book  My Subscription  My Forums  Log Out

AI Acceptance Criteria

 
View related threads: (in this forum | in all forums)

Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [New Releases from Matrix Games] >> World in Flames >> AI Acceptance Criteria Page: [1]
Login
Message << Older Topic   Newer Topic >>
AI Acceptance Criteria - 1/11/2010 10:30:36 AM   
WIF_Killzone

 

Posts: 277
Joined: 4/30/2009
Status: offline
Assuming that "all known bugs are fixed" is one condition that must be met in order to release World in Flames, what conditions must be met for the AI to be deemed ready for release?

For example, would it simply be that the AI functions as designed, with no bugs or are there other play-related conditions that must be met and how could you specify these?
Post #: 1
RE: AI Acceptance Criteria - 1/11/2010 11:36:52 AM   
WIF_Killzone

 

Posts: 277
Joined: 4/30/2009
Status: offline
Forgive me if I am naval gazing, my mind is racing, not a good thing.

The question i think I am asking is how to develop a list of performance indicators or measurement criteria so that Matrix has the right type of information to gage the quality of the AI. i.e. how to determine if its ready for release. This could be be done via a percentage of test cases successful (e.g. We will release the product if 98% of strategic test cases function as designed, etc)

How then will the test cases be developed to test the AI?. In normal software development there would be "UAT" (user acceptance testing) that follows system testing (developer and test team testing). A series of test scenarios comprised of a number of test cases would be created.

An example scenario may be "AI playing Germany attempts to repel invasion of France". A series of pre-conditions would be specified (i.e. France is under german control, Vichy France has been established, germany is at war with Russia, etc, as well as expected results (i.e., Germany does abc, xyz, etc...attempts to repel invasion...deosn't sacrifice russia war to0 much?.. these would all be quite "strategically"broad based scenarios. The test cases then would be more detailed, as well as the pre-conditions, and 'expected results'. Quite a lot of work as you can see. Not to frighten, but I have seen testing budgets as large and larger then requirements, design, and development budgets (for example in banking or insurance where a mistake could prove fatal or cost the company a lot of money) I digress.

Conversely, would test cases simply be created to test and verify the AI's strategic, tactical and operational level "thinking" is working as designed? What would these look like?

I am curious on what the approach will be for testing the AI. Should test planning, scenarios and test case development be started now (if that is the approach that is used), and how could the forum support this work, will additional beta testers be required, has the approach been given any thought yet?

Hmmmmm.

(in reply to WIF_Killzone)
Post #: 2
RE: AI Acceptance Criteria - 1/15/2010 8:17:15 AM   
coregames


Posts: 470
Joined: 8/12/2004
Status: offline

quote:

ORIGINAL: WIF_Killzone

Forgive me if I am naval gazing, my mind is racing, not a good thing.

The question i think I am asking is how to develop a list of performance indicators or measurement criteria so that Matrix has the right type of information to gage the quality of the AI. i.e. how to determine if its ready for release. This could be be done via a percentage of test cases successful (e.g. We will release the product if 98% of strategic test cases function as designed, etc)

How then will the test cases be developed to test the AI?. In normal software development there would be "UAT" (user acceptance testing) that follows system testing (developer and test team testing). A series of test scenarios comprised of a number of test cases would be created.

An example scenario may be "AI playing Germany attempts to repel invasion of France". A series of pre-conditions would be specified (i.e. France is under german control, Vichy France has been established, germany is at war with Russia, etc, as well as expected results (i.e., Germany does abc, xyz, etc...attempts to repel invasion...deosn't sacrifice russia war to0 much?.. these would all be quite "strategically"broad based scenarios. The test cases then would be more detailed, as well as the pre-conditions, and 'expected results'. Quite a lot of work as you can see. Not to frighten, but I have seen testing budgets as large and larger then requirements, design, and development budgets (for example in banking or insurance where a mistake could prove fatal or cost the company a lot of money) I digress.

Conversely, would test cases simply be created to test and verify the AI's strategic, tactical and operational level "thinking" is working as designed? What would these look like?

I am curious on what the approach will be for testing the AI. Should test planning, scenarios and test case development be started now (if that is the approach that is used), and how could the forum support this work, will additional beta testers be required, has the approach been given any thought yet?

Hmmmmm.


If Steve does adopt some form of rubric for 'grading' the performance of the AIO in various situations, in my opinion the grade should be based on a personal assessment rather than some sort of strict formalized criteria. The game is just too big to lend itself to easy quantification, and players (especially experienced WiFers) will have the ability to think at a higher level in order to judge the AIO's performance, which may not be reflected in the outcome of a scenario, depending on luck among other factors.

_____________________________

"The creative combination lays bare the presumption of a lie." -- Lasker

Keith Henderson

(in reply to WIF_Killzone)
Post #: 3
Page:   [1]
All Forums >> [New Releases from Matrix Games] >> World in Flames >> AI Acceptance Criteria Page: [1]
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

1.984