RE: Search arc statistical test (Full Version)

All Forums >> [New Releases from Matrix Games] >> War in the Pacific: Admiral's Edition



Message

Smeulders -> RE: Search arc statistical test (9/16/2010 7:12:03 PM)

I'm a fan of the search arcs as well, so I'd really like for them to have a noticeable impact on my searches. I'd be surprised if anyone ever said that there were no downsides to search arcs though. I'd think it only logical that if you tell your guys explicitly to search only a certain sector, they won't be in any others.  As for the sightings when search arcs were completely the opposite direction, maybe it has something to do with the targets being carriers. Remember, carriers increase their own detection when launching planes, so maybe they aren't the best TFs to use for this test.



rader -> RE: Search arc statistical test (9/16/2010 11:51:18 PM)

Ok, ran the test again at range = 10 hexes. I'm almost reluctant to post this, because these results are strange indeed.

Because someone mentioned CVs have a higher detection level, I used 4 groups of US ships this time. Each groups was composed of 2CA, 1DD, except one was 1CA, 1CL, 1DD. I put all the groups at a range of 10 hexes south of Rabaul like before.

Same group of Nells searching, 30% search/70% rest; 6000ft, searching at range 10. I used 9 samples per case this time.

Results (# TFs spotted)
Case 1. No search arc set (4/4/3/2/1/2/3/2/3) Mean = 2.67
Case 2. "Optimal" search arc (1/1/0/1/0/0/0/1/1) Mean = 0.56
Case 3. "Opposite/worst" search arc (2/1/1/0/1/2/0/1) Mean = 0.89

Clearely in this case, having no search arcs set (0-360) yielded significantly better results than either the optimal or worst search arc cases (p < 0.05).

However, for reasons I can't explain, setting the search arcs to the opposite direction actually improved the searches compared with the optimal search arc case [X(] (although this was not a statistically significant effect, p = 0.15).

What the heck is going on here? [&:]

My only guess is that there was another confounding factor.... maybe I got really bad weather for a large number of the "optimal" search cases? Unfortunately, I did not track this. Maybe there is a strange range effect? Perhaps in the last test, I was getting something strange in the tests because of the CV detection levels? I have no idea; I'm flabergasted. But I'm starting to think setting search arcs is a really bad idea! [8|]

I would appreciate if someone else could run these tests to help shed ligh on this and coroborate the results.







[image]local://upfiles/14041/1945C15360D944CE8474DF0F5BFE4178.jpg[/image]



rader -> RE: Search arc statistical test (9/16/2010 11:54:40 PM)

quote:

ORIGINAL: Mistmatz

I like your test setup, but IMHO the sample size is by far too small too be statistically significant.


That comes out of the t-test. The differences were statistically significant (95% confidence interval) with the given sample size (when p < 0.05). But you're correct that for an accurate model, I would need to include the effects of weather and other differences between the cases. These cases weren't really the same, which does have some bearing on the significance.



CapAndGown -> RE: Search arc statistical test (9/17/2010 12:23:28 AM)


quote:

ORIGINAL: rader

But I'm starting to think setting search arcs is a really bad idea! [8|]




I've been thinking that for a while.



Cad908 -> RE: Search arc statistical test (9/17/2010 1:46:30 AM)

Couple of suggestions:

1. Could you run with advanced weather effects off? I believe this might neutralize weather, did not see if you did this from previous posts.

2. Are you exiting the program completely after each test? The random number generator could be seeding each turn with the same starting point, thus not a true test.

Other than that, you seem to be confirming what many players are seeing in their games.

Thank you for your efforts.



Cad908 -> RE: Search arc statistical test (9/17/2010 1:46:30 AM)

Duplicate deleted



castor troy -> RE: Search arc statistical test (9/17/2010 7:55:25 AM)

quote:

ORIGINAL: rader

Ok, ran the test again at range = 10 hexes. I'm almost reluctant to post this, because these results are strange indeed.

Because someone mentioned CVs have a higher detection level, I used 4 groups of US ships this time. Each groups was composed of 2CA, 1DD, except one was 1CA, 1CL, 1DD. I put all the groups at a range of 10 hexes south of Rabaul like before.

Same group of Nells searching, 30% search/70% rest; 6000ft, searching at range 10. I used 9 samples per case this time.

Results (# TFs spotted)
Case 1. No search arc set (4/4/3/2/1/2/3/2/3) Mean = 2.67
Case 2. "Optimal" search arc (1/1/0/1/0/0/0/1/1) Mean = 0.56
Case 3. "Opposite/worst" search arc (2/1/1/0/1/2/0/1) Mean = 0.89

Clearely in this case, having no search arcs set (0-360) yielded significantly better results than either the optimal or worst search arc cases (p < 0.05).

However, for reasons I can't explain, setting the search arcs to the opposite direction actually improved the searches compared with the optimal search arc case [X(] (although this was not a statistically significant effect, p = 0.15).

What the heck is going on here? [&:]

My only guess is that there was another confounding factor.... maybe I got really bad weather for a large number of the "optimal" search cases? Unfortunately, I did not track this. Maybe there is a strange range effect? Perhaps in the last test, I was getting something strange in the tests because of the CV detection levels? I have no idea; I'm flabergasted. But I'm starting to think setting search arcs is a really bad idea! [8|]

I would appreciate if someone else could run these tests to help shed ligh on this and coroborate the results.


[image]local://upfiles/14041/1945C15360D944CE8474DF0F5BFE4178.jpg[/image]



this example is more in line in what happened in my game too. While your first test actually surprised me about NO archs only being slightly better than focussing all your ac into a small vector, your second test is more in line in what I would have said about my stomach feeling (is this even an English term?).

As it stands now I have done far better without archs as I can´t complain about spotting something at all... without archs...



witpqs -> RE: Search arc statistical test (9/17/2010 8:07:08 AM)

quote:

ORIGINAL: castor troy

this example is more in line in what happened in my game too. While your first test actually surprised me about NO archs only being slightly better than focussing all your ac into a small vector, your second test is more in line in what I would have said about my stomach feeling (is this even an English term?).



Pretty close Castor: "gut feeling"



castor troy -> RE: Search arc statistical test (9/17/2010 8:14:47 AM)


quote:

ORIGINAL: witpqs

quote:

ORIGINAL: castor troy

this example is more in line in what happened in my game too. While your first test actually surprised me about NO archs only being slightly better than focussing all your ac into a small vector, your second test is more in line in what I would have said about my stomach feeling (is this even an English term?).



Pretty close Castor: "gut feeling"



thanks [:)]



Apollo11 -> RE: Search arc statistical test (9/17/2010 8:33:58 AM)

Hi all,

BTW, how did you do your testings?

Did you use original scenario (without changing anything) or you created your own scenario based on "Coral Sea" scenario?

How did you run your test (i.e. did you load WitP-AE and then load the scenario and tested, closed the scenario, loaded the scenario and tested again or you closed the WitP-AE as well)?

When I tested in UV and WitP the rule was always to close the program (i.e. UV, WitP) in between every single test!!!


Leo "Apollo11"



Sardaukar -> RE: Search arc statistical test (9/17/2010 8:35:08 AM)

Might be worth doing couple more tests and then post on Tech Support, since it seems that search arcs are working exactly opposite as intended, which would suggest it's a bug. 



tigercub -> RE: Search arc statistical test (9/17/2010 10:35:58 AM)

i have been wasting a lot of time on those ACKs..at least we know about it now....please post to Tech Support to get the ball rolling...thanks guys for bringing this to light of Day.

Tigercub!



d0mbo -> RE: Search arc statistical test (9/17/2010 11:04:44 AM)

If the search arcs aren't working as intended, is it safe to say the same applies to the ASW arcs?

If so, i have not only wasted time putting my emily's and netties on search arcs, but MANY LB/FP groups on ASW duty as well!

Hope a Dev can alleviate my (our?) fears!




Sardaukar -> RE: Search arc statistical test (9/17/2010 11:25:57 AM)

I was wondering why AI was mauling my subs with planes..and I was not hitting much with my ASW...this might be the reason...[:(]



olperfessor -> Search by surface ships' floatplanes (9/17/2010 12:40:17 PM)

If setting the search arc is pointless, I can stop wishing that we could set the arc relative to a ship's direction (I wanted my TFs to be able to do ASW searches ahead of them, even when their courses changed after reaching waypoints.

Then again, I recall reading that search aircraft conduct searches within four hexes of their bases regardless of where (or whether) search arcs are specified. Not sure if this is the case.



Sardaukar -> RE: Search by surface ships' floatplanes (9/17/2010 12:47:41 PM)


quote:

ORIGINAL: olperfessor


Then again, I recall reading that search aircraft conduct searches within four hexes of their bases regardless of where (or whether) search arcs are specified. Not sure if this is the case.


In addition to 4 hexes, they actually do search further too. Just that probability of spotting should be more probable withing arc and less outside. Unfortunately it doesn't seem to be the case if these tests are true.

In other words, if lucky and system worked as designed, there would be a chance of spotting outside the arc depending on range just as inside arc, but it'd be considerable lower.



Chickenboy -> RE: Search by surface ships' floatplanes (9/17/2010 2:07:16 PM)

rader, et. al.,

Very interesting tests and discussion. I'm following with interest. Thanks for doing this...



Walloc -> RE: Search by surface ships' floatplanes (9/17/2010 2:30:08 PM)

Thx for doing the tests rader.

If u per Sardurkar advice do more tests. It might interrestíng if it could be done in an aircraft with radar vs one with out.
PBY-5 vs PBY-5A comes to mind, as apparently ppl been getting better results with non radar equiped planes, then those with radar. Contrary to what one should think.

Kind regards,

Rasmus



TR Shrum -> RE: Search by surface ships' floatplanes (9/17/2010 2:41:31 PM)

I have been concerned about setting Archs for Search and ASW and the Results of same for some time now. After reading these posts I went and check several of my In Progress games against the AI. I have been confused and frustrated by the poor results recieved by my AC when setting Archs and how much better the AI's results seem to be. I assumed it had something to do with the fact the Game Engine has to know something about where my units are in order to make things work. So I loaded several Saved Games in Two Player Mode and to my surprise I found that the AI never sets Archs. In every case (ASW or Search) no Archs are set, Max Range is set, and the % of AC assigned is either 33% or 66%; the balance at Rest. I've now ran several Turns of my most recent game without setting Archs of any kind for my uints and BINGO, I'm now getting much better Search and ASW contacts.



morganbj -> RE: Search arc statistical test (9/17/2010 3:56:07 PM)


quote:

ORIGINAL: Mistmatz

I like your test setup, but IMHO the sample size is by far too small too be statistically significant.

Well, larger samples would be nice, but the t test accounts for sample size through the use of degrees of freedom. It calculates the actual probabilities based on sample size. If the calculate probability (p) is below some value, usually .05, then it is statistically significant. So, I have to disagree with you there, Mistmatz.

But, this p is true only for the conditions set for the test. I think to draw the kind of conclusions you want, the test(s) would have to be set up with a little more control. E.g., I think each TF would have to be the exact same makup of ships. Several tests would then have to be conducted with varying detection distances, pilot experience and skill levels, and so forth. Then the same tests run with different TF makeup. Please understand, Rader, that what you did is not wrong, it's actually guite good. It's just not the complete picture - yet.

The thing to remember is that statistical significance does not mean practical significance. What does a .14 increase in probability really mean? One additional TF will get discovered 14 times every 100 searches. On average. And with TF makeup as it appeared in the test. And with the exact same range, with the same pilots, etc. That is why a little more testting should be done. And perhaps, Mistmatz, this is really what you were referring to, so I do not mean to discount your concern with my comments above.

This test did get significant results that should not be ignored as evidence that something may be a tad odd.

Nice job.


BTW, I teach statistics at the doctoral level at a university and really appreciate someone who uses these techniques for simple things like this. They don't always need to be used for some overblown, academic, pompus, research-about-nothing-important dissertation. I'm quite impressed, Rader. Drive on.



Buck Beach -> RE: Search by surface ships' floatplanes (9/17/2010 4:23:21 PM)


quote:

ORIGINAL: Chickenboy

rader, et. al.,

Very interesting tests and discussion. I'm following with interest. Thanks for doing this...


I too am following the issue closely.

The search arcs, ASW searchs and pilot trainings are all very cool features and any problems need be addressed an fixed.



SteveD64 -> RE: Search by surface ships' floatplanes (9/17/2010 6:45:10 PM)

I've been setting search arcs and now that I know that they're not as effective I'm actually relieved that I won't have to micromanage them anymore.  Heresy I know!  Maybe when this is fixed there can be a pregame option to have effective search arcs or keep it the way it is now.



hunchback77 -> RE: Search arc statistical test (9/17/2010 7:03:22 PM)

Rader, can you run the test with the Nells searching at 100% and no rest at 2000ft, same arc. I have a feeling that 30% Search/ 70% Rest may be causing the problems or it could be the 6000ft altitude setting. Thank you.



witpqs -> RE: Search by surface ships' floatplanes (9/17/2010 7:41:13 PM)


quote:

ORIGINAL: Walloc

Thx for doing the tests rader.

If u per Sardurkar advice do more tests. It might interrestíng if it could be done in an aircraft with radar vs one with out.
PBY-5 vs PBY-5A comes to mind, as apparently ppl been getting better results with non radar equiped planes, then those with radar. Contrary to what one should think.

Kind regards,

Rasmus


IIRC the only difference there is wheels - strictly flying boat versus amphibious. Try the PBY-4 versus the PBY-5.



witpqs -> RE: Search by surface ships' floatplanes (9/17/2010 7:42:22 PM)

Let's give them a bit to investigate this now that some tests have been run.



Q-Ball -> RE: Search by surface ships' floatplanes (9/17/2010 8:06:42 PM)

Great work on this. I am very interested in the results.

I would love to see similar work on ASW, which is even more opaque, and difficult to replicate. I suppose that would take a long time, since hits can be random, but it would be great to finally know how to maximize ASW.



Walloc -> RE: Search by surface ships' floatplanes (9/17/2010 10:50:13 PM)


quote:

ORIGINAL: witpqs


quote:

ORIGINAL: Walloc

Thx for doing the tests rader.

If u per Sardurkar advice do more tests. It might interrestíng if it could be done in an aircraft with radar vs one with out.
PBY-5 vs PBY-5A comes to mind, as apparently ppl been getting better results with non radar equiped planes, then those with radar. Contrary to what one should think.

Kind regards,

Rasmus


IIRC the only difference there is wheels - strictly flying boat versus amphibious. Try the PBY-4 versus the PBY-5.


Yes, my bad PBY-4 it is.



ade670 -> RE: Search by surface ships' floatplanes (9/17/2010 11:57:49 PM)

Guys

I'm a little frustrated with the lack of definitive answers from matrix on this .

Let's just keep things simple - forget altitude forget these rather spurious settings - search arcs dont work as intended or as described in the game manual - that = broken - please fix

Whilst I enjoy this game I'm increasingly of the opinion it is far from polished
On the information in the manual I have acted to set search arcs in my games - I want my valuable hours back !!!!

Thanks for the hard work so far highlighting this - the devs should send u a cheque



CapAndGown -> RE: Search by surface ships' floatplanes (9/18/2010 1:24:35 AM)


quote:

ORIGINAL: ade670

Guys

I'm a little frustrated with the lack of definitive answers from matrix on this .

Let's just keep things simple - forget altitude forget these rather spurious settings - search arcs dont work as intended or as described in the game manual - that = broken - please fix

Whilst I enjoy this game I'm increasingly of the opinion it is far from polished
On the information in the manual I have acted to set search arcs in my games - I want my valuable hours back !!!!

Thanks for the hard work so far highlighting this - the devs should send u a cheque


I think this is uncalled for. The devs are doing a very good job. This is an extremely complex game. Of course there will be problems here and there. This problem is hardly a game killer. In fact, there are no game killing bugs I know of. (I would like to get the radar thingy fixed fast, though. [;)]) So chill out. With their limited resources they have to prioritize, but I am sure that this problem will be addressed in a reasonable amount of time.



stuman -> RE: Search by surface ships' floatplanes (9/18/2010 1:28:12 AM)


quote:

ORIGINAL: ade670

Guys

I'm a little frustrated with the lack of definitive answers from matrix on this .

Let's just keep things simple - forget altitude forget these rather spurious settings - search arcs dont work as intended or as described in the game manual - that = broken - please fix

Whilst I enjoy this game I'm increasingly of the opinion it is far from polished
On the information in the manual I have acted to set search arcs in my games - I want my valuable hours back !!!!

Thanks for the hard work so far highlighting this - the devs should send u a cheque


We need to be patient. Just because we have raised an issue in the last 2 days does not mean that the Devs have had enough time to see this thread, drop everything they are doing in life, run all tests, etc and give us an answer.

And good on you Rader for doing some research. One of the many reasons this is one of the best games out there, and one of the best forum groups I have ever encountered.



Page: <<   < prev  1 [2] 3 4 5   next >   >>

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
3.515625