Leo's ASW TEST (and scenario) - problems spotted! (Full Version)

All Forums >> [Current Games From Matrix.] >> [World War II] >> War In The Pacific - Struggle Against Japan 1941 - 1945



Message

Apollo11 -> Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 4:18:48 PM)

Hi all,

Here is Leo's ASW TEST (and scenario) made from scratch in new WitP v1.70 BETA (i.e. with OOB of v1.60 since after that no OOB changes were done).

I wanted to do this for log time bi only today managed to get some free time to do it...


Download Link:

Leo ASW TEST Scenario #120.zip (10-16-2005)


The ASW TEST scenario is made to explore various tactics and possibilities!


NOTE:
All ships in scenario have their default EXP rating whilst all air units have their EXP at 75 and morale at 75 (you can change those in scenario editor rather easy if you wish).


Possible serious issue with Air ASW search

I did found one rather disturbing fact already...

The ASW air search is _EXTREMELY_ effective and out of 36 submarines 33 were discovered.

IMHO this is way way way exaggerated and needs to be toned down...


Please look at the following pictures:

[img]http://free-zg.htnet.hr/Leonardo_Rogic/Images/Air_ASW_01.gif[/img]

[img]http://free-zg.htnet.hr/Leonardo_Rogic/Images/Air_ASW_02.gif[/img]

[img]http://free-zg.htnet.hr/Leonardo_Rogic/Images/Air_ASW_03.gif[/img]


Leo "Apollo11"



rtrapasso -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 4:35:03 PM)

quote:

Possible serious issue with Air ASW search

I did found one rather disturbing fact already...

The ASW air search is _EXTREMELY_ effective and out of 36 submarines 33 were discovered.

IMHO this is way way way exaggerated and needs to be toned down...


This has been a problem for some time now (and many a bitter complaint has graced the forum about it) - it did NOT originate with this patch. Alas, AFAIK, they did not say they were going to address the problem in this patch - just how FAST you could sink the subs was issue addressed.



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 4:49:28 PM)

Hi all,

quote:

ORIGINAL: rtrapasso

This has been a problem for some time now (and many a bitter complaint has graced the forum about it) - it did NOT originate with this patch. Alas, AFAIK, they did not say they were going to address the problem in this patch - just how FAST you could sink the subs was issue addressed.


I know... it was set several WitP versions ago (the WitP v1.70 BETA only chnaged ship based ASW)...


But, unfortunately, nobody did comprehensive test to actually see how good/bad Air ASW search is until I just did it!


Leo "Apollo11"



aletoledo -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 5:10:26 PM)

72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.

perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 5:16:39 PM)

Hi all,

I just spotted "tiny" mistake in my TEST ASW scenario - all TF slots had "9999" delay set so no new TF were able to create - I fixed this and uploaded the fixed file insetad of old one

Please note that I left the file name and version same (in order not to confuse users more).

So, if you downloaded my scenario in past 60 minutes (since itw as out) please reload - sorry for inconvinience...


Leo "Apollo11"



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 5:22:44 PM)

Hi all,

quote:

ORIGINAL: aletoledo

72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.

perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.


The scenario is available for all so all kind of tests can be made (with all kind of alterations).


BTW, I on purpose set 100% ASW in order to get best possible ASW Air search case.


Also note that 72 aircraft is not much if you look how big area covered with submarines is. We are talking about _HUGE_ piece of sea - the submarines roughly take 60 (sixty) HEXes of deep sea and each HEX is 60x69 nautical miles!!!

60x60x60 nmi = 216000 square nautical miles!


Leo "Apollo11"



jwilkerson -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:10:46 PM)

Since July 2004, there have been many comments that subs are way too detectable by air search - thus far M2b3 have chosen other means to address the overall issue of subs sinking too fast - not by making subs less detectable - but by changing the way the attacks work - and I think again in 1.7 this has been the approach. Not sure why search hasn't been addressed but must be because it is thought to be more diffcult to make the change from that direction.




dereck -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:17:33 PM)


quote:

ORIGINAL: aletoledo

72 recon planes doing ASW at 100% seems like an extravagense that would never have occured in real life.

perhaps if you had 3-4 planes on ASW (one of the chutai's at 40%) and the rest on normal naval search (again not 100%) and it did this, I would agree otherwise IMO you can't try to complain about a tactic inside a game (setting everything to 100% ASW) and then claim that its not the same in real life, where it wasn't performed like that.


True. If you're going to run a test make the test with a reasonable level of planes instead of going overboard like this and other "tests" that you've run. Going overboard you're just ensuring you get off-kilter results.



madmickey -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:18:43 PM)

Ron is Right



rtrapasso -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:27:42 PM)


quote:

ORIGINAL: madmickey

Ron is Right

[:D][:D]



Nomad -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:28:25 PM)


quote:

ORIGINAL: madmickey

Ron is Right


We know that, now convince Matrix Games and 2x3 of that. [:D]



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:29:17 PM)

Hi all,

quote:

ORIGINAL: dereck

If you're going to run a test make the test with a reasonable level of planes instead of going overboard like this and other "tests" that you've run. Going overboard you're just ensuring you get off-kilter results.


Please explain what you mean under "test" I have made?

What is "overboard" about this and other test I made?

Who else made any other tests worth wile that were available to general WitP population (an what tests you made so that you can complain about what I did)?


BTW, do you really think that putting 72 aircraft (and out of those 36 are biplanes) on AWS is overkill?

Have you read the AARs (in those number on ASW aircraft can be counted in hundreds)?


Leo "Apollo11"



dereck -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:54:12 PM)

Yes I think your testing is flawed. What you're basically doing is overkill and ensuring that if the ASW search routine IS changed it's going to be based on flawed testing and force people in future version to have MORE than your 72 planes on ASW just to be able to spot submarines.

If you want to do a honest and decent test have three levels of searching. A low value, a medium value and a high value - in both number of planes used AND the amount assigned to ASW search. You'd be doing a lot of tests but that way you have a RANGE of spotting instead of a high value which will obviously result in sub spottings.

If, as you say, people in AARs are using hundresds of planes in ASW patrol then YES they're going to spot submarines. That is obvious - all your test did was to show the obvious.

You want to do a serious test vary the test data so if it's the number of planes, the percent assigned to ASW, a combination of both, or an unrelated variable can be determined. It could show that 72 planes at 10% ASW may do just as well as 18 planes at 100% for example. You may also want to run the same tests as both the Japanese and Allied to see if the results are the same or different.

As it is your test is only showing one side and, in my opinion, not objective enough to really be taken seriously since you can have people like me questioning it.

And as far as testing, I've done plenty of program and system testing and test plan writting in my 16 years in IT to know that you have to be objective and plan for a lot of possibilities - you've only planned and tested for one at most.



jwilkerson -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 6:58:38 PM)

Unfortunately in some of the games I've played you wouldn't get into overboard until you'd gotten up towards 500+ planes !!! ( for better or worse, if we give the players a loop hole - they will exploit it !!! House rules have subsequently fixed in that game )





Nomad -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 7:09:49 PM)

I don't have a lot of time for testing but since Leo gave us a good tool to use I tried a few tests. I took the bettys off of any search. Put the Mavis on 100% Naval search at 6000 feet. and the two 9 aircraft float planes on 100% ASW at 1000 ft. I think this is pretty standard( it would be for me ). In three runs I got 25, 22, 21 subs detected. So, anyone want to venture a guess on what the detected number should be? [:)]



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 7:19:43 PM)

Hi all,

quote:

ORIGINAL: dereck

Yes I think your testing is flawed. What you're basically doing is overkill and ensuring that if the ASW search routine IS changed it's going to be based on flawed testing and force people in future version to have MORE than your 72 planes on ASW just to be able to spot submarines.

If you want to do a honest and decent test have three levels of searching. A low value, a medium value and a high value - in both number of planes used AND the amount assigned to ASW search. You'd be doing a lot of tests but that way you have a RANGE of spotting instead of a high value which will obviously result in sub spottings.

If, as you say, people in AARs are using hundresds of planes in ASW patrol then YES they're going to spot submarines. That is obvious - all your test did was to show the obvious.

You want to do a serious test vary the test data so if it's the number of planes, the percent assigned to ASW, a combination of both, or an unrelated variable can be determined. It could show that 72 planes at 10% ASW may do just as well as 18 planes at 100% for example. You may also want to run the same tests as both the Japanese and Allied to see if the results are the same or different.

As it is your test is only showing one side and, in my opinion, not objective enough to really be taken seriously since you can have people like me questioning it.

And as far as testing, I've done plenty of program and system testing and test plan writting in my 16 years in IT to know that you have to be objective and plan for a lot of possibilities - you've only planned and tested for one at most.


Have you read what I wrote when I first published this?

I think not..


I asked that WitP community (i.e. all of us who are interested in better WitP) test and check for _THEMSELVES_!

I _ALWAYS_ post scenario and _NOT_ results _ONLY_ (i.e. where scenario is "hidden" or unavailable so the results can't be verified).

I also wrote that I made one test - more testing can be done by me (when I get time) or by any other interested WitP player who thinks current way of WitP ASW is not ideal.

You can test if you want to help.


BTW, I also have 15+ years in IT and know what testing is - testing (in essence) is:

- controlled environment (this we have - I made scenario that can be tested and tested)
- inclusion of all possible variations and alterations (this we have - I made scenario that can be changed easily)
- need of many runs to establish pattern


So who/what is flawed here?


Leo "Apollo11"



dereck -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 7:26:44 PM)

Testing SHOULD be in a controlled environment Leo so you can have results that will hold water. By doing multiple tests using something like the grid below you could show both a pattern and/or trend and the results couldn't be called skewered by people.



[image]local://upfiles/13978/037CE53C801644B78F946BE3443E1109.jpg[/image]



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 7:33:07 PM)

Hi all,

quote:

ORIGINAL: dereck

Testing SHOULD be in a controlled environment Leo so you can have results that will hold water. By doing multiple tests using something like the grid below you could show both a pattern and/or trend and the results couldn't be called skewered by people.

[image]local://upfiles/13978/037CE53C801644B78F946BE3443E1109.jpg[/image]


I agree 100% - I always agreed (and I always run several test runs with each test being reload of scenario in order to get new "seed" each time - just saving game and reloading save is wrong for testing because you will always have same "seed").


So... who volunteers to help to see what is really going with WitP Air and ASW search?


Leo "Apollo11"



Apollo11 -> Another discovery (now proven fact)... (10/16/2005 7:35:50 PM)

Hi all,

Another discovery (now proven fact)...

In ASW Air search the range is halved (just as it is written in manual).

So if you set range (via dialer) to 6 HEXes you get ASW search to 3 HEXes out!


Leo "Apollo11"



jwilkerson -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 7:51:36 PM)


quote:

ORIGINAL: Nomad

I don't have a lot of time for testing but since Leo gave us a good tool to use I tried a few tests. I took the bettys off of any search. Put the Mavis on 100% Naval search at 6000 feet. and the two 9 aircraft float planes on 100% ASW at 1000 ft. I think this is pretty standard( it would be for me ). In three runs I got 25, 22, 21 subs detected. So, anyone want to venture a guess on what the detected number should be? [:)]


Well if there was a " move to station " mode ( sub operating mostly on the surface in waters believed to be mostly free of enemy air search while in route to patrol area ) ... and a "patrol mode" ... submarine mostly submerged during the day to avoid air search ... and the above searches were with the sub in patrol mode and made without radar .. and during the day ... then the detected number should be some small number between about 0 and 2 ... ( even sub commanders make mistakes and surface sometimes when they shouldn't .. or they have engine trouble ... or they see a juicey target .. or they go aground ... )




DFalcon -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 8:21:04 PM)


quote:

ORIGINAL: Apollo11


So... who volunteers to help to see what is really going with WitP Air and ASW search?


Leo "Apollo11"



I will run some tests and post results.



Apollo11 -> Relationship between selected HEX range and enemy submarines sighted... (10/16/2005 8:38:08 PM)

Hi all,

Relationship between selected HEX range (remember it is halved!) and enemy submarines sighted...


According to test is appears that lowering the range in order to increase "density" of search aircraft over some area is not working (i.e. idea that with less range the search will be better because there is less area covered with same number of aircraft used).

So... unless some other factors are in play (like enemy base in range with strong CAP that routinely kills your search aircraft) there is no need to lower the range from MAX...


Leo "Apollo11"

[image]local://upfiles/3416/9BD0E91CC76241F889234C3250276898.gif[/image]



jwilkerson -> RE: Relationship between selected HEX range and enemy submarines sighted... (10/16/2005 8:50:45 PM)

Not to overly complexify the effort - but Naval Search is also extremely effective at killing subs ( at leasts in prior to 1.7 releases ) ... in fact many players hardly use ASW Search since Naval Search gets the job done so well. Try bombers like Martin or PBM at 1000 foot at normal range ... glug glug go the subs ...




Apollo11 -> Relationship between altitude and enemy submarines sighted... (10/16/2005 9:05:09 PM)

Hi all,

Relationship between altitude and enemy submarines sighted...


According to test is appears that altitude isn't much affecting submarine spotting.

So... unless some other factors are in play (like enemy flak on submarines in area that can damage your search aircraft or wish to attack more accurately during search if opportunity arises) there is no need to lower the altitude...


Leo "Apollo11"


[image]local://upfiles/3416/635B4A436C094014844EB43C4BDF6A32.gif[/image]


P.S. [Edit]
Typo (last 0 was "lost") - last two numbers in table should be "10000ft" and "20000ft"!



DFalcon -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 9:37:22 PM)

The first test is complete using Apollo test scenario.

For this test the search aircraft were set as follows.

Emily 6000ft Nav. Search 90%
Betty 6000ft Nav. Attack 20%
Jake 6000ft Nav. Search 90%
Alf stood down.

The US subs were placed around the base 6 at each range from 1 to 6. (See screen shot) and moved each turn maintaining distance from the base to eliminate DL. I ran 10 turns and recorded the results. A brief summary is below.

Average number of Subs spotted each turn 7.7, overall chance of discovery 21%.

Weather ? ? ? Overcast Rain Overcast Rain Rain Overcast Partial
Turn 1 2 3 4 5 6 7 8 9 10 %
Range
1 3 5 4 1 1 2 1 2 1 1 35.00%
2 3 2 0 0 4 1 1 2 1 2 26.67%
3 1 0 2 0 1 1 0 2 1 0 13.33%
4 4 2 1 2 0 2 1 0 2 1 25.00%
5 2 1 0 1 0 2 1 0 2 2 18.33%
6 1 1 2 0 0 1 0 1 0 0 10.00%
Total 14 11 9 4 6 9 4 7 7 6 21.39%
% 38.89% 30.56% 25.00% 11.11% 16.67% 25.00% 11.11% 19.44% 19.44% 16.67%
Overall % 21.39%
Average 7.7


[image]local://upfiles/14523/E1FBF69EB28540878D467136D548898C.jpg[/image]



Apollo11 -> RE: Relationship between altitude and enemy submarines sighted... (10/16/2005 9:48:24 PM)

Hi all,

Relationship between aircraft used and enemy submarines sighted...


According to test is appears that number of aircraft used scale rather good and linear with submarine spotting.

So... more aircraft used equals better spotting!


NOTE: there were 36 enemy submarines altogether.


Leo "Apollo11"


[image]local://upfiles/3416/D883294FED42422F8D24F2D9D1666659.gif[/image]



jwilkerson -> RE: Relationship between altitude and enemy submarines sighted... (10/16/2005 9:51:00 PM)


quote:

ORIGINAL: Apollo11

<snip>

According to test is appears that altitude isn't much affecting submarine spotting.

So... unless some other factors are in play (like enemy flak on submarines in area that can damage your search aircraft or wish to attack more accurately during search if opportunity arises) there is no need to lower the altitude...

<snip>



(A) Of course altitude SHOULD affect submarine spotting ( or any ship spotting ) ... while wakes can be seen far a great distance ... telling what kind of ship is making the wake one must get closer and lower. Per Chez ( who looked for subs for a living ) sub search altitude woul be 3,000 ft IRL. But we hear you - in the game - spotting the sub is apparently independent of altitude.

(B) Seeing the sub may be the only thing you're testing - but "killing" or at least "attacking" and "hitting" the sub does seem to happen much more often when the altitude is lower ( 1,000 feet ) in my experience - but maybe that is another set of tests.




Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 9:51:47 PM)

Hi all,

quote:

ORIGINAL: DFalcon

The first test is complete using Apollo test scenario.

For this test the search aircraft were set as follows.

Emily 6000ft Nav. Search 90%
Betty 6000ft Nav. Attack 20%
Jake 6000ft Nav. Search 90%
Alf stood down.

The US subs were placed around the base 6 at each range from 1 to 6. (See screen shot) and moved each turn maintaining distance from the base to eliminate DL. I ran 10 turns and recorded the results. A brief summary is below.

Average number of Subs spotted each turn 7.7, overall chance of discovery 21%.

Weather ? ? ? Overcast Rain Overcast Rain Rain Overcast Partial
Turn 1 2 3 4 5 6 7 8 9 10 %
Range
1 3 5 4 1 1 2 1 2 1 1 35.00%
2 3 2 0 0 4 1 1 2 1 2 26.67%
3 1 0 2 0 1 1 0 2 1 0 13.33%
4 4 2 1 2 0 2 1 0 2 1 25.00%
5 2 1 0 1 0 2 1 0 2 2 18.33%
6 1 1 2 0 0 1 0 1 0 0 10.00%
Total 14 11 9 4 6 9 4 7 7 6 21.39%
% 38.89% 30.56% 25.00% 11.11% 16.67% 25.00% 11.11% 19.44% 19.44% 16.67%
Overall % 21.39%
Average 7.7


[image]local://upfiles/14523/E1FBF69EB28540878D467136D548898C.jpg[/image]


Thanks "DFalcon" (and nice modification)!


BTW, if you use "code" you can get tables to sort much better - here is what it would look like: 

Average number of Subs spotted each turn 7.7, overall chance of discovery 21%.

Weather	?	?	?	Overcast	Rain	Overcast	Rain	Rain	Overcast	Partial	
Turn	1	2	3	4	5	6	7	8	9	10	%
Range											
1	3	5	4	1	1	2	1	2	1	1	35.00%
2	3	2	0	0	4	1	1	2	1	2	26.67%
3	1	0	2	0	1	1	0	2	1	0	13.33%
4	4	2	1	2	0	2	1	0	2	1	25.00%
5	2	1	0	1	0	2	1	0	2	2	18.33%
6	1	1	2	0	0	1	0	1	0	0	10.00%
Total	14	11	9	4	6	9	4	7	7	6	21.39%
%	38.89%	30.56%	25.00%	11.11%	16.67%	25.00%	11.11%	19.44%	19.44%	16.67%	
Overall %	21.39%										
Average	7.7



Leo "Apollo11"



Nomad -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 9:54:17 PM)

Thanks Leo, that made his much easier to read. We learn something everyday.[:D]



Apollo11 -> RE: Leo's ASW TEST (and scenario) - problems spotted! (10/16/2005 10:00:46 PM)

Hi all,

quote:

ORIGINAL: Nomad

Thanks Leo, that made his much easier to read. We learn something everyday.[:D]


No problem at all - glad to be of help!


BTW, why I didn't think of your way of arranging submarines... [;)]


Leo "Apollo11"



Page: [1] 2 3   next >   >>

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
1.75