RE: Search arc statistical test

Matrix Games Forums

Forums Register Login Photo Gallery Member List Search Calendars FAQ

My Profile Inbox Address Book My Subscription My Forums Log Out

RE: Search arc statistical test

View related threads: (in this forum | in all forums)

Logged in as: Guest

Users viewing this topic: none

Printable Version

All Forums >> [New Releases from Matrix Games] >> War in the Pacific: Admiral's Edition >> RE: Search arc statistical test

Page: << < prev 1 [2] 3 4 5 next > >>

Message

<< Older Topic Newer Topic >>

RE: Search arc statistical test - 9/16/2010 7:12:03 PM

Smeulders

Posts: 1879
Joined: 8/9/2009
Status: offline

I'm a fan of the search arcs as well, so I'd really like for them to have a noticeable impact on my searches. I'd be surprised if anyone ever said that there were no downsides to search arcs though. I'd think it only logical that if you tell your guys explicitly to search only a certain sector, they won't be in any others. As for the sightings when search arcs were completely the opposite direction, maybe it has something to do with the targets being carriers. Remember, carriers increase their own detection when launching planes, so maybe they aren't the best TFs to use for this test.

(in reply to Lomri)

Post #: 31

RE: Search arc statistical test - 9/16/2010 11:51:18 PM

rader

Posts: 1238
Joined: 9/13/2004
Status: offline

Ok, ran the test again at range = 10 hexes. I'm almost reluctant to post this, because these results are strange indeed.

Because someone mentioned CVs have a higher detection level, I used 4 groups of US ships this time. Each groups was composed of 2CA, 1DD, except one was 1CA, 1CL, 1DD. I put all the groups at a range of 10 hexes south of Rabaul like before.

Same group of Nells searching, 30% search/70% rest; 6000ft, searching at range 10. I used 9 samples per case this time.

Results (# TFs spotted)
Case 1. No search arc set (4/4/3/2/1/2/3/2/3) Mean = 2.67
Case 2. "Optimal" search arc (1/1/0/1/0/0/0/1/1) Mean = 0.56
Case 3. "Opposite/worst" search arc (2/1/1/0/1/2/0/1) Mean = 0.89

Clearely in this case, having no search arcs set (0-360) yielded significantly better results than either the optimal or worst search arc cases (p < 0.05).

However, for reasons I can't explain, setting the search arcs to the opposite direction actually improved the searches compared with the optimal search arc case

(although this was not a statistically significant effect, p = 0.15).

What the heck is going on here?

My only guess is that there was another confounding factor.... maybe I got really bad weather for a large number of the "optimal" search cases? Unfortunately, I did not track this. Maybe there is a strange range effect? Perhaps in the last test, I was getting something strange in the tests because of the CV detection levels? I have no idea; I'm flabergasted. But I'm starting to think setting search arcs is a really bad idea!

I would appreciate if someone else could run these tests to help shed ligh on this and coroborate the results.

Attachment (1)

< Message edited by rader -- 9/16/2010 11:52:56 PM >

(in reply to Smeulders)

Post #: 32

RE: Search arc statistical test - 9/16/2010 11:54:40 PM

rader

Posts: 1238
Joined: 9/13/2004
Status: offline

quote:

ORIGINAL: Mistmatz

I like your test setup, but IMHO the sample size is by far too small too be statistically significant.

That comes out of the t-test. The differences were statistically significant (95% confidence interval) with the given sample size (when p < 0.05). But you're correct that for an accurate model, I would need to include the effects of weather and other differences between the cases. These cases weren't really the same, which does have some bearing on the significance.

< Message edited by rader -- 9/16/2010 11:57:01 PM >

(in reply to Mistmatz)

Post #: 33

RE: Search arc statistical test - 9/17/2010 12:23:28 AM

CapAndGown

Posts: 3206
Joined: 3/6/2001
From: Virginia, USA
Status: offline

quote:

ORIGINAL: rader

But I'm starting to think setting search arcs is a really bad idea!

I've been thinking that for a while.

(in reply to rader)

Post #: 34

RE: Search arc statistical test - 9/17/2010 1:46:30 AM

Cad908

Posts: 1333
Joined: 10/9/2009
Status: offline

Couple of suggestions:

1. Could you run with advanced weather effects off? I believe this might neutralize weather, did not see if you did this from previous posts.

2. Are you exiting the program completely after each test? The random number generator could be seeding each turn with the same starting point, thus not a true test.

Other than that, you seem to be confirming what many players are seeing in their games.

Thank you for your efforts.

(in reply to Smeulders)

Post #: 35

RE: Search arc statistical test - 9/17/2010 1:46:30 AM

Cad908

Posts: 1333
Joined: 10/9/2009
Status: offline

Duplicate deleted

< Message edited by Cad908 -- 9/17/2010 1:47:54 AM >

(in reply to Smeulders)

Post #: 36

RE: Search arc statistical test - 9/17/2010 7:55:25 AM

castor troy

Posts: 14330
Joined: 8/23/2004
From: Austria
Status: offline

quote:

ORIGINAL: rader

Ok, ran the test again at range = 10 hexes. I'm almost reluctant to post this, because these results are strange indeed.

Because someone mentioned CVs have a higher detection level, I used 4 groups of US ships this time. Each groups was composed of 2CA, 1DD, except one was 1CA, 1CL, 1DD. I put all the groups at a range of 10 hexes south of Rabaul like before.

Same group of Nells searching, 30% search/70% rest; 6000ft, searching at range 10. I used 9 samples per case this time.

Results (# TFs spotted)
Case 1. No search arc set (4/4/3/2/1/2/3/2/3) Mean = 2.67
Case 2. "Optimal" search arc (1/1/0/1/0/0/0/1/1) Mean = 0.56
Case 3. "Opposite/worst" search arc (2/1/1/0/1/2/0/1) Mean = 0.89

Clearely in this case, having no search arcs set (0-360) yielded significantly better results than either the optimal or worst search arc cases (p < 0.05).

However, for reasons I can't explain, setting the search arcs to the opposite direction actually improved the searches compared with the optimal search arc case (although this was not a statistically significant effect, p = 0.15).

What the heck is going on here?

My only guess is that there was another confounding factor.... maybe I got really bad weather for a large number of the "optimal" search cases? Unfortunately, I did not track this. Maybe there is a strange range effect? Perhaps in the last test, I was getting something strange in the tests because of the CV detection levels? I have no idea; I'm flabergasted. But I'm starting to think setting search arcs is a really bad idea!

I would appreciate if someone else could run these tests to help shed ligh on this and coroborate the results.

this example is more in line in what happened in my game too. While your first test actually surprised me about NO archs only being slightly better than focussing all your ac into a small vector, your second test is more in line in what I would have said about my stomach feeling (is this even an English term?).

As it stands now I have done far better without archs as I can�t complain about spotting something at all... without archs...

< Message edited by castor troy -- 9/17/2010 7:56:32 AM >

_____________________________

(in reply to rader)

Post #: 37

RE: Search arc statistical test - 9/17/2010 8:07:08 AM

witpqs

Posts: 26087
Joined: 10/4/2004
From: Argleton
Status: offline

quote:

ORIGINAL: castor troy

this example is more in line in what happened in my game too. While your first test actually surprised me about NO archs only being slightly better than focussing all your ac into a small vector, your second test is more in line in what I would have said about my stomach feeling (is this even an English term?).

Pretty close Castor: "gut feeling"

(in reply to castor troy)

Post #: 38

RE: Search arc statistical test - 9/17/2010 8:14:47 AM

castor troy

Posts: 14330
Joined: 8/23/2004
From: Austria
Status: offline

quote:

ORIGINAL: witpqs

quote:

ORIGINAL: castor troy

this example is more in line in what happened in my game too. While your first test actually surprised me about NO archs only being slightly better than focussing all your ac into a small vector, your second test is more in line in what I would have said about my stomach feeling (is this even an English term?).

Pretty close Castor: "gut feeling"

thanks

_____________________________

(in reply to witpqs)

Post #: 39

RE: Search arc statistical test - 9/17/2010 8:33:58 AM

Apollo11

Posts: 24082
Joined: 6/7/2001
From: Zagreb, Croatia
Status: offline

Hi all,

BTW, how did you do your testings?

Did you use original scenario (without changing anything) or you created your own scenario based on "Coral Sea" scenario?

How did you run your test (i.e. did you load WitP-AE and then load the scenario and tested, closed the scenario, loaded the scenario and tested again or you closed the WitP-AE as well)?

When I tested in UV and WitP the rule was always to close the program (i.e. UV, WitP) in between every single test!!!

Leo "Apollo11"

_____________________________

Prior Preparation & Planning Prevents Pathetically Poor Performance!

A & B: WitW, WitE, WbtS, GGWaW, GGWaW2-AWD, HttR, CotA, BftB, CF
P: UV, WitP, WitP-AE

(in reply to rader)

Post #: 40

RE: Search arc statistical test - 9/17/2010 8:35:08 AM

Sardaukar

Posts: 9847
Joined: 11/28/2001
From: Finland/Israel
Status: offline

Might be worth doing couple more tests and then post on Tech Support, since it seems that search arcs are working exactly opposite as intended, which would suggest it's a bug.

_____________________________

"To meaningless French Idealism, Liberty, Fraternity and Equality...we answer with German Realism, Infantry, Cavalry and Artillery" -Prince von B�lov, 1870-

(in reply to castor troy)

Post #: 41

RE: Search arc statistical test - 9/17/2010 10:35:58 AM

tigercub

Posts: 2004
Joined: 2/3/2003
From: brisbane oz
Status: offline

i have been wasting a lot of time on those ACKs..at least we know about it now....please post to Tech Support to get the ball rolling...thanks guys for bringing this to light of Day.

Tigercub!

_____________________________

You have enemies? Good. That means you've stood up for something, sometime in your life

(in reply to Sardaukar)

Post #: 42

RE: Search arc statistical test - 9/17/2010 11:04:44 AM

d0mbo

Posts: 592
Joined: 8/21/2009
From: Holland
Status: offline

If the search arcs aren't working as intended, is it safe to say the same applies to the ASW arcs?

If so, i have not only wasted time putting my emily's and netties on search arcs, but MANY LB/FP groups on ASW duty as well!

Hope a Dev can alleviate my (our?) fears!

< Message edited by d0mbo -- 9/17/2010 12:06:18 PM >

(in reply to tigercub)

Post #: 43

RE: Search arc statistical test - 9/17/2010 11:25:57 AM

Sardaukar

Posts: 9847
Joined: 11/28/2001
From: Finland/Israel
Status: offline

I was wondering why AI was mauling my subs with planes..and I was not hitting much with my ASW...this might be the reason...

_____________________________

"To meaningless French Idealism, Liberty, Fraternity and Equality...we answer with German Realism, Infantry, Cavalry and Artillery" -Prince von B�lov, 1870-

(in reply to d0mbo)

Post #: 44

Search by surface ships' floatplanes - 9/17/2010 12:40:17 PM

olperfessor

Posts: 98
Joined: 4/28/2003
From: New York, N.Y.
Status: offline

If setting the search arc is pointless, I can stop wishing that we could set the arc relative to a ship's direction (I wanted my TFs to be able to do ASW searches ahead of them, even when their courses changed after reaching waypoints.

Then again, I recall reading that search aircraft conduct searches within four hexes of their bases regardless of where (or whether) search arcs are specified. Not sure if this is the case.

(in reply to Smeulders)

Post #: 45

RE: Search by surface ships' floatplanes - 9/17/2010 12:47:41 PM

Sardaukar

Posts: 9847
Joined: 11/28/2001
From: Finland/Israel
Status: offline

quote:

ORIGINAL: olperfessor

Then again, I recall reading that search aircraft conduct searches within four hexes of their bases regardless of where (or whether) search arcs are specified. Not sure if this is the case.

In addition to 4 hexes, they actually do search further too. Just that probability of spotting should be more probable withing arc and less outside. Unfortunately it doesn't seem to be the case if these tests are true.

In other words, if lucky and system worked as designed, there would be a chance of spotting outside the arc depending on range just as inside arc, but it'd be considerable lower.

_____________________________

"To meaningless French Idealism, Liberty, Fraternity and Equality...we answer with German Realism, Infantry, Cavalry and Artillery" -Prince von B�lov, 1870-

(in reply to olperfessor)

Post #: 46

RE: Search by surface ships' floatplanes - 9/17/2010 2:07:16 PM

Chickenboy

Posts: 24520
Joined: 6/29/2002
From: San Antonio, TX
Status: offline

rader, et. al.,

Very interesting tests and discussion. I'm following with interest. Thanks for doing this...

_____________________________

(in reply to Sardaukar)

Post #: 47

RE: Search by surface ships' floatplanes - 9/17/2010 2:30:08 PM

Walloc

Posts: 3141
Joined: 10/30/2006
From: Denmark
Status: offline

Thx for doing the tests rader.

If u per Sardurkar advice do more tests. It might interrest�ng if it could be done in an aircraft with radar vs one with out.
PBY-5 vs PBY-5A comes to mind, as apparently ppl been getting better results with non radar equiped planes, then those with radar. Contrary to what one should think.

Kind regards,

Rasmus

(in reply to Chickenboy)

Post #: 48

RE: Search by surface ships' floatplanes - 9/17/2010 2:41:31 PM

TR Shrum

Posts: 74
Joined: 2/13/2006
Status: offline

I have been concerned about setting Archs for Search and ASW and the Results of same for some time now. After reading these posts I went and check several of my In Progress games against the AI. I have been confused and frustrated by the poor results recieved by my AC when setting Archs and how much better the AI's results seem to be. I assumed it had something to do with the fact the Game Engine has to know something about where my units are in order to make things work. So I loaded several Saved Games in Two Player Mode and to my surprise I found that the AI never sets Archs. In every case (ASW or Search) no Archs are set, Max Range is set, and the % of AC assigned is either 33% or 66%; the balance at Rest. I've now ran several Turns of my most recent game without setting Archs of any kind for my uints and BINGO, I'm now getting much better Search and ASW contacts.

(in reply to Walloc)

Post #: 49

RE: Search arc statistical test - 9/17/2010 3:56:07 PM

morganbj

Posts: 3634
Joined: 8/12/2007
From: Mosquito Bite, Texas
Status: offline

quote:

ORIGINAL: Mistmatz

I like your test setup, but IMHO the sample size is by far too small too be statistically significant.

Well, larger samples would be nice, but the t test accounts for sample size through the use of degrees of freedom. It calculates the actual probabilities based on sample size. If the calculate probability (p) is below some value, usually .05, then it is statistically significant. So, I have to disagree with you there, Mistmatz.

But, this p is true only for the conditions set for the test. I think to draw the kind of conclusions you want, the test(s) would have to be set up with a little more control. E.g., I think each TF would have to be the exact same makup of ships. Several tests would then have to be conducted with varying detection distances, pilot experience and skill levels, and so forth. Then the same tests run with different TF makeup. Please understand, Rader, that what you did is not wrong, it's actually guite good. It's just not the complete picture - yet.

The thing to remember is that statistical significance does not mean practical significance. What does a .14 increase in probability really mean? One additional TF will get discovered 14 times every 100 searches. On average. And with TF makeup as it appeared in the test. And with the exact same range, with the same pilots, etc. That is why a little more testting should be done. And perhaps, Mistmatz, this is really what you were referring to, so I do not mean to discount your concern with my comments above.

This test did get significant results that should not be ignored as evidence that something may be a tad odd.

Nice job.

BTW, I teach statistics at the doctoral level at a university and really appreciate someone who uses these techniques for simple things like this. They don't always need to be used for some overblown, academic, pompus, research-about-nothing-important dissertation. I'm quite impressed, Rader. Drive on.

(in reply to Mistmatz)

Post #: 50

RE: Search by surface ships' floatplanes - 9/17/2010 4:23:21 PM

Buck Beach

Posts: 1973
Joined: 6/25/2000
From: Upland,CA,USA
Status: offline

quote:

ORIGINAL: Chickenboy

rader, et. al.,

Very interesting tests and discussion. I'm following with interest. Thanks for doing this...

I too am following the issue closely.

The search arcs, ASW searchs and pilot trainings are all very cool features and any problems need be addressed an fixed.

(in reply to Chickenboy)

Post #: 51

RE: Search by surface ships' floatplanes - 9/17/2010 6:45:10 PM

SteveD64

Posts: 570
Joined: 10/26/2006
From: Shaker Hts, Ohio, USA
Status: offline

I've been setting search arcs and now that I know that they're not as effective I'm actually relieved that I won't have to micromanage them anymore. Heresy I know! Maybe when this is fixed there can be a pregame option to have effective search arcs or keep it the way it is now.

(in reply to Buck Beach)

Post #: 52

RE: Search arc statistical test - 9/17/2010 7:03:22 PM

hunchback77

Posts: 189
Joined: 5/13/2002
From: Whitby, Ontario, Canada
Status: offline

Rader, can you run the test with the Nells searching at 100% and no rest at 2000ft, same arc. I have a feeling that 30% Search/ 70% Rest may be causing the problems or it could be the 6000ft altitude setting. Thank you.

(in reply to rader)

Post #: 53

RE: Search by surface ships' floatplanes - 9/17/2010 7:41:13 PM

witpqs

Posts: 26087
Joined: 10/4/2004
From: Argleton
Status: offline

quote:

ORIGINAL: Walloc

Thx for doing the tests rader.

If u per Sardurkar advice do more tests. It might interrest�ng if it could be done in an aircraft with radar vs one with out.
PBY-5 vs PBY-5A comes to mind, as apparently ppl been getting better results with non radar equiped planes, then those with radar. Contrary to what one should think.

Kind regards,

Rasmus

IIRC the only difference there is wheels - strictly flying boat versus amphibious. Try the PBY-4 versus the PBY-5.

(in reply to Walloc)

Post #: 54

RE: Search by surface ships' floatplanes - 9/17/2010 7:42:22 PM

witpqs

Posts: 26087
Joined: 10/4/2004
From: Argleton
Status: offline

Let's give them a bit to investigate this now that some tests have been run.

(in reply to witpqs)

Post #: 55

RE: Search by surface ships' floatplanes - 9/17/2010 8:06:42 PM

Q-Ball

Posts: 7336
Joined: 6/25/2002
From: Chicago, Illinois
Status: offline

Great work on this. I am very interested in the results.

I would love to see similar work on ASW, which is even more opaque, and difficult to replicate. I suppose that would take a long time, since hits can be random, but it would be great to finally know how to maximize ASW.

_____________________________

(in reply to witpqs)

Post #: 56

RE: Search by surface ships' floatplanes - 9/17/2010 10:50:13 PM

Walloc

Posts: 3141
Joined: 10/30/2006
From: Denmark
Status: offline

quote:

ORIGINAL: witpqs

quote:

ORIGINAL: Walloc

Thx for doing the tests rader.

If u per Sardurkar advice do more tests. It might interrest�ng if it could be done in an aircraft with radar vs one with out.
PBY-5 vs PBY-5A comes to mind, as apparently ppl been getting better results with non radar equiped planes, then those with radar. Contrary to what one should think.

Kind regards,

Rasmus

IIRC the only difference there is wheels - strictly flying boat versus amphibious. Try the PBY-4 versus the PBY-5.

Yes, my bad PBY-4 it is.

(in reply to witpqs)

Post #: 57

RE: Search by surface ships' floatplanes - 9/17/2010 11:57:49 PM

ade670

Posts: 68
Joined: 12/1/2009
Status: offline

Guys

I'm a little frustrated with the lack of definitive answers from matrix on this .

Let's just keep things simple - forget altitude forget these rather spurious settings - search arcs dont work as intended or as described in the game manual - that = broken - please fix

Whilst I enjoy this game I'm increasingly of the opinion it is far from polished
On the information in the manual I have acted to set search arcs in my games - I want my valuable hours back !!!!

Thanks for the hard work so far highlighting this - the devs should send u a cheque

(in reply to Walloc)

Post #: 58

RE: Search by surface ships' floatplanes - 9/18/2010 1:24:35 AM

CapAndGown

Posts: 3206
Joined: 3/6/2001
From: Virginia, USA
Status: offline

quote:

ORIGINAL: ade670

Guys

I'm a little frustrated with the lack of definitive answers from matrix on this .

Let's just keep things simple - forget altitude forget these rather spurious settings - search arcs dont work as intended or as described in the game manual - that = broken - please fix

Whilst I enjoy this game I'm increasingly of the opinion it is far from polished
On the information in the manual I have acted to set search arcs in my games - I want my valuable hours back !!!!

Thanks for the hard work so far highlighting this - the devs should send u a cheque

I think this is uncalled for. The devs are doing a very good job. This is an extremely complex game. Of course there will be problems here and there. This problem is hardly a game killer. In fact, there are no game killing bugs I know of. (I would like to get the radar thingy fixed fast, though.

) So chill out. With their limited resources they have to prioritize, but I am sure that this problem will be addressed in a reasonable amount of time.

(in reply to ade670)

Post #: 59

RE: Search by surface ships' floatplanes - 9/18/2010 1:28:12 AM

stuman

Posts: 3907
Joined: 9/14/2008
From: Elvis' Hometown
Status: offline

quote:

ORIGINAL: ade670

Guys

I'm a little frustrated with the lack of definitive answers from matrix on this .

Let's just keep things simple - forget altitude forget these rather spurious settings - search arcs dont work as intended or as described in the game manual - that = broken - please fix

Whilst I enjoy this game I'm increasingly of the opinion it is far from polished
On the information in the manual I have acted to set search arcs in my games - I want my valuable hours back !!!!

Thanks for the hard work so far highlighting this - the devs should send u a cheque

We need to be patient. Just because we have raised an issue in the last 2 days does not mean that the Devs have had enough time to see this thread, drop everything they are doing in life, run all tests, etc and give us an answer.

And good on you Rader for doing some research. One of the many reasons this is one of the best games out there, and one of the best forum groups I have ever encountered.

_____________________________

" Gentlemen, you can't fight in here! This is the War Room. " President Muffley