Search arc statistical test

Search arc statistical test (Full Version)

All Forums >> [New Releases from Matrix Games] >> War in the Pacific: Admiral's Edition

Message

rader -> Search arc statistical test (9/16/2010 4:17:42 AM)
I've always been diligent about trying to set search arcs. They seem like a neat feature, and appeal to my micromanagement side. However, after reading the thread about search arcs, I decided the run a test. Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s. The Nells are looking for the 4 US TFs, all 14 hexes from Rabaul (see picture). They are the default ones from Coral Sea scenario. The TFs are relatively small; 2 have a CV and a few CAs/DDs each; 2 have an AO and DD (or maybe ~two AO each + DD(s)). Anyway, somewhat hard targets to spot. Comparison of 3 cases, 7 samples each: 1. 7 samples of Nells with no search arcs set (0-360). 2. 7 samples of Nells set to search "optimal" search arc, covering exactly the 40 degrees the US TFs are aligned along at 14 hexes. This is about as good a search as you could do if you knew exactly where the enemy was (apart from sending more planes!). 3. 7 samples of Nells set to opposite search arcs (looking north in exactly the wrong direction, around 340-020 deg). Results (number of TFs spoted per sample): No search arcs: (1/1/1/2/1/2/0) (mean = 1.14 TFs spotted) "Optimal" search arc: (0/2/1/1/1/2/0) (mean = 1 TF spotted) Wrong (opposite) search arc: (0/0/1/0/0/0/0) (mean = 0.14 TF spotted) I ran two 2 sample t-tests in SYSTAT to compare cases (1. & 2.), and (1. & 3.) No search arcs (1) vs. Optimal search arc (2). p = 0.73. (no significance) Optimal search arc vs. wrong search arc. p < 0.006 (quite significant difference). Discussion: I don't claim this covers all cases, and there are confounding effects like weather which I didn't examine. Maybe in some cases you do get better results by setting the right search arcs (longer/shorter search distance/altitude/larger TFs, etc.). And although the mean is slightly lower, I certainly can't claim that setting the right search arc is better or worse than setting none at all. But I do think this demonstrates that setting the wrong search arc is worse than setting the right one OR setting none at all. So in this type of situation you probably aren't much worse off setting no search arc than setting the right one. Which, of course, you can't perfectly predict. Conclusion: On average (assuming the enemy sometimes comes from directions you don't set as arcs), you will find him more often under similar conditions to this test, if you don't set arcs. I think I'm going to take off all my search arcs. [image]local://upfiles/14041/5074023A6D44495680A4094BB41C733B.jpg[/image]

stuman -> RE: Search arc statistical test (9/16/2010 4:29:53 AM)
Hmm.

Lrfss -> RE: Search arc statistical test (9/16/2010 4:34:21 AM)
Starting to think the same thing myself!

witpqs -> RE: Search arc statistical test (9/16/2010 4:40:51 AM)
...which would mean it's broken. Either too good results without arcs, or not good enough results with arcs.

Cribtop -> RE: Search arc statistical test (9/16/2010 5:17:46 AM)
Given the huge player time investment, setting search arcs should provide a meaningful benefit.

witpqs -> RE: Search arc statistical test (9/16/2010 5:21:33 AM)
quote: ORIGINAL: Cribtop Given the huge player time investment, setting search arcs should provide a meaningful benefit. Search arcs are meant to be more realistic. Setting no arcs, IIRC, means "choose the search paths vectors randomly". As the player knows where the likely threat is, search arcs should yield significantly more real contacts that using randomly selected vectors. From the anecdotes of those who have tried both (I've only used arcs in AE), and from the test that rader performed, it looks like there is a program problem.

crsutton -> RE: Search arc statistical test (9/16/2010 5:27:46 AM)
I abandoned search arcs about 90 game turns ago. I find the results just as good with no search arc set and you can cover more territory. Your tests seem to back this up. I would think some more testing is called for.

Sardaukar -> RE: Search arc statistical test (9/16/2010 5:44:44 AM)
Run the tests with enemy TFs bit closer to base. I seem to recall, that you rarely spot anything more than 12 hexes away or so. This may significantly change the results.

Misconduct -> RE: Search arc statistical test (9/16/2010 5:45:39 AM)
quote: ORIGINAL: crsutton I abandoned search arcs about 90 game turns ago. I find the results just as good with no search arc set and you can cover more territory. Your tests seem to back this up. I would think some more testing is called for. Agreed, I have yet to set a "search" arc in 4 games and have yet had a problem with spotting. Biggest problem is setting the arc and forgetting about it tends to annoy me having to go back and switch it, so I just leave everything on 80% search, 20% rest.

rader -> RE: Search arc statistical test (9/16/2010 5:46:39 AM)
quote: ORIGINAL: Sardaukar Run the tests with enemy TFs bit closer to base. I seem to recall, that you rarely spot anything more than 12 hexes away or so. This may significantly change the results. Good idea, will do that. How far would you suggest? 10 hexes?

Sardaukar -> RE: Search arc statistical test (9/16/2010 6:20:06 AM)
quote: ORIGINAL: rader quote: ORIGINAL: Sardaukar Run the tests with enemy TFs bit closer to base. I seem to recall, that you rarely spot anything more than 12 hexes away or so. This may significantly change the results. Good idea, will do that. How far would you suggest? 10 hexes? 8-10 should be fine.

jomni -> RE: Search arc statistical test (9/16/2010 6:59:18 AM)
This is the reason why I don't manually set the arcs. An unncesary nuisance just like pilot training.

Mistmatz -> RE: Search arc statistical test (9/16/2010 7:14:02 AM)
I like your test setup, but IMHO the sample size is by far too small too be statistically significant.

castor troy -> RE: Search arc statistical test (9/16/2010 7:37:41 AM)
shouldn´t NO search archs be far worse than with your search archs of only 1/9 of the area covered with archs. Yet, even when the Nells have to cover 9 times more area they have spotted 14% more in your test. [&:]

Apollo11 -> RE: Search arc statistical test (9/16/2010 8:02:45 AM)
Hi all, quote: ORIGINAL: rader I've always been diligent about trying to set search arcs. They seem like a neat feature, and appeal to my micromanagement side. However, after reading the thread about search arcs, I decided the run a test. Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s. The Nells are looking for the 4 US TFs, all 14 hexes from Rabaul (see picture). They are the default ones from Coral Sea scenario. The TFs are relatively small; 2 have a CV and a few CAs/DDs each; 2 have an AO and DD (or maybe ~two AO each + DD(s)). Anyway, somewhat hard targets to spot. Comparison of 3 cases, 7 samples each: 1. 7 samples of Nells with no search arcs set (0-360). 2. 7 samples of Nells set to search "optimal" search arc, covering exactly the 40 degrees the US TFs are aligned along at 14 hexes. This is about as good a search as you could do if you knew exactly where the enemy was (apart from sending more planes!). 3. 7 samples of Nells set to opposite search arcs (looking north in exactly the wrong direction, around 340-020 deg). Results (number of TFs spoted per sample): No search arcs: (1/1/1/2/1/2/0) (mean = 1.14 TFs spotted) "Optimal" search arc: (0/2/1/1/1/2/0) (mean = 1 TF spotted) Wrong (opposite) search arc: (0/0/1/0/0/0/0) (mean = 0.14 TF spotted) I ran two 2 sample t-tests in SYSTAT to compare cases (1. & 2.), and (1. & 3.) No search arcs (1) vs. Optimal search arc (2). p = 0.73. (no significance) Optimal search arc vs. wrong search arc. p < 0.006 (quite significant difference). Discussion: I don't claim this covers all cases, and there are confounding effects like weather which I didn't examine. Maybe in some cases you do get better results by setting the right search arcs (longer/shorter search distance/altitude/larger TFs, etc.). And although the mean is slightly lower, I certainly can't claim that setting the right search arc is better or worse than setting none at all. But I do think this demonstrates that setting the wrong search arc is worse than setting the right one OR setting none at all. So in this type of situation you probably aren't much worse off setting no search arc than setting the right one. Which, of course, you can't perfectly predict. Conclusion: On average (assuming the enemy sometimes comes from directions you don't set as arcs), you will find him more often under similar conditions to this test, if you don't set arcs. I think I'm going to take off all my search arcs. [image]local://upfiles/14041/5074023A6D44495680A4094BB41C733B.jpg[/image] Very nice test - thanks! [:)] BTW, I did some extensive research in UV and WitP regarding search but didn't have time to do it in WitP-AE... Anyway here are my old threads (please click on links below): 10/16/2005 Leo's ASW TEST (and scenario) - problems spotted! 11/21/2005 Leo's air ASW Search TEST WitP v1.795... In those I discovered that in "Search" every airplane is actually flying each HEX it covers and that in each HEX the crew has a chance of discovery (depending on various other variables like altitude, weather, skill etc.)... quote: Here is how I think it works: #1 Every pilot in every squadron assigned for air search ("Naval Search" or "ASW") that passes all pre-flight checks (enough pilots, enough aircraft, leader check, percentage of aircraft assigned to mission, weather etc.) individually flies the "spiral" (i.e. concentric circles with increased range) up to the range user selected (halved in case of ASW). #2 The "spiral" (i.e. concentric circles with increased range) covers every single HEX (in range) and then checks are made in every such HEX for discovery and countermeasures (enemy CAP, AA etc.). What worries me now is that even with opposite search (i.e. searching nortn in your test) there were some discoveries. If the routine works similarly as it used to work in WitP then only the northern HEXes and not at all in the south... hmmm... BTW, did you stand down all other assets that can do the search? Leo "Apollo11"

castor troy -> RE: Search arc statistical test (9/16/2010 8:30:15 AM)
quote: ORIGINAL: Apollo11 Hi all, quote: ORIGINAL: rader I've always been diligent about trying to set search arcs. They seem like a neat feature, and appeal to my micromanagement side. However, after reading the thread about search arcs, I decided the run a test. Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s. The Nells are looking for the 4 US TFs, all 14 hexes from Rabaul (see picture). They are the default ones from Coral Sea scenario. The TFs are relatively small; 2 have a CV and a few CAs/DDs each; 2 have an AO and DD (or maybe ~two AO each + DD(s)). Anyway, somewhat hard targets to spot. Comparison of 3 cases, 7 samples each: 1. 7 samples of Nells with no search arcs set (0-360). 2. 7 samples of Nells set to search "optimal" search arc, covering exactly the 40 degrees the US TFs are aligned along at 14 hexes. This is about as good a search as you could do if you knew exactly where the enemy was (apart from sending more planes!). 3. 7 samples of Nells set to opposite search arcs (looking north in exactly the wrong direction, around 340-020 deg). Results (number of TFs spoted per sample): No search arcs: (1/1/1/2/1/2/0) (mean = 1.14 TFs spotted) "Optimal" search arc: (0/2/1/1/1/2/0) (mean = 1 TF spotted) Wrong (opposite) search arc: (0/0/1/0/0/0/0) (mean = 0.14 TF spotted) I ran two 2 sample t-tests in SYSTAT to compare cases (1. & 2.), and (1. & 3.) No search arcs (1) vs. Optimal search arc (2). p = 0.73. (no significance) Optimal search arc vs. wrong search arc. p < 0.006 (quite significant difference). Discussion: I don't claim this covers all cases, and there are confounding effects like weather which I didn't examine. Maybe in some cases you do get better results by setting the right search arcs (longer/shorter search distance/altitude/larger TFs, etc.). And although the mean is slightly lower, I certainly can't claim that setting the right search arc is better or worse than setting none at all. But I do think this demonstrates that setting the wrong search arc is worse than setting the right one OR setting none at all. So in this type of situation you probably aren't much worse off setting no search arc than setting the right one. Which, of course, you can't perfectly predict. Conclusion: On average (assuming the enemy sometimes comes from directions you don't set as arcs), you will find him more often under similar conditions to this test, if you don't set arcs. I think I'm going to take off all my search arcs. [image]local://upfiles/14041/5074023A6D44495680A4094BB41C733B.jpg[/image] Very nice test - thanks! [:)] BTW, I did some extensive research in UV and WitP regarding search but didn't have time to do it in WitP-AE... Anyway here are my old threads (please click on links below): 10/16/2005 Leo's ASW TEST (and scenario) - problems spotted! 11/21/2005 Leo's air ASW Search TEST WitP v1.795... In those I discovered that in "Search" every airplane is actually flying each HEX it covers and that in each HEX the crew has a chance of discovery (depending on various other variables like altitude, weather, skill etc.)... quote: Here is how I think it works: #1 Every pilot in every squadron assigned for air search ("Naval Search" or "ASW") that passes all pre-flight checks (enough pilots, enough aircraft, leader check, percentage of aircraft assigned to mission, weather etc.) individually flies the "spiral" (i.e. concentric circles with increased range) up to the range user selected (halved in case of ASW). #2 The "spiral" (i.e. concentric circles with increased range) covers every single HEX (in range) and then checks are made in every such HEX for discovery and countermeasures (enemy CAP, AA etc.). What worries me now is that even with opposite search (i.e. searching nortn in your test) there were some discoveries. If the routine works similarly as it used to work in WitP then only the northern HEXes and not at all in the south... hmmm... BTW, did you stand down all other assets that can do the search? Leo "Apollo11" he did: Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s.

Apollo11 -> RE: Search arc statistical test (9/16/2010 9:10:38 AM)
Hi all, quote: ORIGINAL: castor troy he did: Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s. Christian, yep... I read that in original first post... but I was wondering about all other assets as well... [;)] Leo "Apollo11"

d0mbo -> RE: Search arc statistical test (9/16/2010 10:19:55 AM)
Inquiring minds follow this thread and hope to hear a DEV's opinion on this. If I don't have to bother with search arcs (or at least: it doesn't give an distinct advantage in doing so), that would be good to know!

jomni -> RE: Search arc statistical test (9/16/2010 10:22:55 AM)
I would rather hear them say that it will give only a slight advantage (as the way it is working now) in order not to make it a requirement for all players. If people claim that it is broken and the devs should make it work... this will penalize those people who don't bother setting search arcs like me. Same sentiments with pilot training.

Sardaukar -> RE: Search arc statistical test (9/16/2010 10:23:11 AM)
quote: ORIGINAL: Apollo11 What worries me now is that even with opposite search (i.e. searching nortn in your test) there were some discoveries. If the routine works similarly as it used to work in WitP then only the northern HEXes and not at all in the south... hmmm... BTW, did you stand down all other assets that can do the search? Leo "Apollo11" Devs (I think it was michaelm) eplained that search arcs are supposed to increase chance of detection within arc (and most likely decrease it outside it). Setting arc does not prevent spotting ourside arc, but it should concentrate search more inside arc.

FatR -> RE: Search arc statistical test (9/16/2010 1:26:56 PM)
I recently started abandoning search arcs as an experiment, using small 9-plane units flying ASW in coastal areas where enemy subs made their presence known. I haven't seen any decrease in detections so far and am gradually adopting "no search arcs" policy for all units.

ade670 -> RE: Search arc statistical test (9/16/2010 1:45:31 PM)
Guys, 3 questions if I may: 1. IF I am on a coastal hex with 50% of the search arc on land and 50% search arc at sea - eg Townsville, will the AI automatically dismiss the 50% land element when plotting naval or asw search arcs. I have always presumed that the routine would still plot both search arcs over the land mass hence the requirement to use the manual search arcs. 2. In the case of small sqdns 4 or less - surely setting up search arcs in a specific direction has got to improve the chances of spotting by if nothing else removing the odds of a the planes flying in the opposite direction to that which is required. 3. Finally, if a report of possible sighting is mention in the Ops report, by a search sqn on 'auto search', will the AI search script automatically fly the same radials on the next turn in order to improve the detection? Any help with this increasingly grey area would be helpful. Ade

Sredni -> RE: Search arc statistical test (9/16/2010 2:32:09 PM)
Well I guess this simplifies naval search setup heh. I wonder if it also effects asw. Though I think we'll still want to setup search arcs for areas where we want to keep our search planes from overflying heavily defended bases. (I have a search plane hub near singapore in singakawan??. one group searching a zone right up to singapore to the northwest and another group flying farther north into the sea east of singapore. singapore has a heavy fighter defense I don't want to overfly) I'm kinda disappointed to be honest. I like micromanaging search arcs. I like to press the Z key and see all the pretty search zones show up heh.

crsutton -> RE: Search arc statistical test (9/16/2010 3:42:37 PM)
quote: ORIGINAL: Sredni Well I guess this simplifies naval search setup heh. I wonder if it also effects asw. Though I think we'll still want to setup search arcs for areas where we want to keep our search planes from overflying heavily defended bases. (I have a search plane hub near singapore in singakawan??. one group searching a zone right up to singapore to the northwest and another group flying farther north into the sea east of singapore. singapore has a heavy fighter defense I don't want to overfly) I'm kinda disappointed to be honest. I like micromanaging search arcs. I like to press the Z key and see all the pretty search zones show up heh. I am using no arcs for ASW and it seems to be doing just fine. Skill level and the range set for the ASW force seems to be more of a factor. The longer the range the more diluted the effort.

Canoerebel -> RE: Search arc statistical test (9/16/2010 3:50:37 PM)
After seeing plausible comments from many other players to this effect months ago, I discontinued setting search arcs in my game with Miller. At that point, we were well into 1944 in our game. For my game with Q-Ball, I haven't set an arc of any kind - search or ASW. As best I can tell my results are absolutely fine. Conclusion: I'm very glad to do away with the micromanaging hassle of setting arcs.

castor troy -> RE: Search arc statistical test (9/16/2010 3:54:30 PM)
quote: ORIGINAL: Canoerebel After seeing plausible comments from many other players to this effect months ago, I discontinued setting search arcs in my game with Miller. At that point, we were well into 1944 in our game. For my game with Q-Ball, I haven't set an arc of any kind - search or ASW. As best I can tell my results are absolutely fine. Conclusion: I'm very glad to do away with the micromanaging hassle of setting arcs. I found the micromanagement great, with the assumption that it would be of course a huge benefit to have 12 Catalinas searching only 90 degrees instead of 12 Catalinas searching 360 degrees. Problem was, the enemy bombarded Akyab probably a dozen times without being spotted once while having dozens of patrols focussed exactly where the enemy was going in and out all the time. That made me more than sceptical and that was when I threw the archs overboard, it worked just fine with me ever since. To be honest, I was amazed about the test even getting that many spottings WITH archs. I´m not surprised it got more without archs, even though it should achieve far more spottings with archs than without.

witpqs -> RE: Search arc statistical test (9/16/2010 4:09:05 PM)
quote: ORIGINAL: ade670 1. IF I am on a coastal hex with 50% of the search arc on land and 50% search arc at sea - eg Townsville, will the AI automatically dismiss the 50% land element when plotting naval or asw search arcs. I have always presumed that the routine would still plot both search arcs over the land mass hence the requirement to use the manual search arcs. No, it will search the whole thing.

Shark7 -> RE: Search arc statistical test (9/16/2010 4:55:55 PM)
quote: ORIGINAL: Cribtop Given the huge player time investment, setting search arcs should provide a meaningful benefit. Well you do get some benefit, you don't waste time searchin over-land...but from these results that is it. [8\|]

CapAndGown -> RE: Search arc statistical test (9/16/2010 6:30:57 PM)
I will just add to the chorus that I too have given up on search arcs. I was becoming rather skeptical already when I read Castor Troy's comments a while back and decided to give it a whirl. Sure enough, my results have been just fine if not better without the arcs. This would be fine with me, except as one poster noted, there are times when you would like your planes to NOT search some area where the CAP is very heavy. Yet, I am now worried that using search arcs in this way will actually degrade my ability to spot ships.

Lomri -> RE: Search arc statistical test (9/16/2010 7:05:25 PM)
The part that is most striking about this test is the "wrong direction" conclusions compared to "no search arch". I tried googling to find the thread (with no luck), but I recall a developer saying that there was no negative side to having a search arch defined. You'd get better results in your search arch but no negative results outside of. This test, while a smallish test, still shows a huge negative to spotting outside an arch. IF there was no downside and only upside to setting archs, I'd still use them. But if the gut feeling of folks and the limited testing shows that search archs are on par or worse, then search arches are a net-negative. Too bad too, I do get a kick out of the Z key showing my search coverage as I try to plug gaps.

Page: [1] 2 3 4 5 next > >>

Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

0.765625