|
KnightHawk75 -> RE: [Logged] Crashing JASSM & JASSM-ER (1/30/2021 6:23:25 PM)
|
I spent some time trying investigating this. I've seen this before especially when moving missiles, it's sporadic though (just because you don't get it on one startup doesn't mean you will not on another), and once it starts or is around it stays around, and a game restart is required to clear up for me when it does happen. What caught my attention is it was in this same area the op's scene is in where I first experienced this last March or April, and once or twice since, and seemingly only in this area, and by area I mean 40-90N,20-60E.
TLDR: Very weird indeed!
Non TLDR:
As for 15x vs other timing I don't think that's it entirely. The _really tricky_ part of this problem, at least for me|my system, is being unable to constantly repo it on a fresh startup. It either happens "activates\triggers" or it doesn't, and on average for me it's only about 1:10 for me, maybe 1:8 if I flush standby ram first (though who knows if that really helped and I didn't just get lucky a few times).
Using the OP's scene I could identify when it was "active" or not and replicate it. It required doing nothing other than reloading the map in the editor from the recent files list. Also when active the issue will stay around if you load another scene (test rig attached with far less going on) or the OP's again, or create a new one,etc.
When the problem showed itself I started watching the if (calculated-unitAGL < 0) then crash the missile into the ground calc. As you might expect the problem was AGL turning up negative during lots of these compares (the gui had correct values till the point of it happening, as did the activeunit ASL wise it seemed). I was expecting to see small negatives, in amounts roughly equal to the "elevation jitter" across a flight path, figured if a thread was 'running behind' or something in updating or something, that could account for a small negative and crash the weapon.
That's _not_ what I saw though, instead yes sometimes they were small, but sometimes they were confusingly massive, even when something passed the < 0 check often the value evaluated was wildly wrong. One extreme example was one that actually passed the check, with an AGL of ~32kft (having been launched at ~42k), yet the 158A was actually flying at ~400ft and 60ft AGL (for an hour mind you as I kept refueling it cause it was one out 96 that refused to die I wanted to know what made it special) and the objects other properties said the same fly at ~400ft. It's like something told it you're asl is 32k and then flagged it to not be updated again (separate from what the gui was pulling from). Another head-scratch was on a 154-C1 launched over the sea at a land target, when it crashed over land for example the AGL in the calc was -2108 feet, when the land below it was ~100ft and elevation changes around it for next mile were ~0-50ft, and there was never anything it flew over higher than about 500 ft. The water it traveled over 10's of miles prior never exceeded ~-1500. Very strange.
I got lucky a couple times and happen to have one selected while it crashed, like 1/2 second before this it was reading fine, then suddenly -1783 agl in 1/2 a second.
[img]https://i.postimg.cc/Q9x7gBHq/Putin-Surge-Caught-In-Act.jpg[/img]
Here's just another caught in the act (as you can see by age I kept him around awhile.)
[img]https://i.postimg.cc/Z9KtBK06/Putin-Surge-Caught-In-Act2.jpg[/img]
A third dying a second after I moved it from land to sea, which had a habit of happening during that game instance:
[img]https://i.postimg.cc/XrCCC4G1/Putin-Surge-Caught-In-Act3.jpg[/img]
An area where 99% of the time anything moved to it or launched through it died in a few seconds pretty darn reliably (but only on one startup instance - though I was trying this area because it's where I previously had problems ~6mo ago.):
[img]https://i.postimg.cc/vDTLg5G0/Putin-Surge-Sea-Of-Death.jpg[/img]
Later plopped a b1 in there and launched 48 158'c, all 48 died in about 10 seconds or before exiting that area. I mention this area or water more generally because it'was interesting to watch missles decend to what should be ~30ft, but at around XXX they would insta-pop back up to ~2500, then head down again, insta-pop back up, rinse and repeat till they die. This happens when the problem is active over land too but watching over water is easier than land as there should be few if no corrections for changing elevation of the landscape that might explain movement. Weirder still is a test scene I made for that area didn't demonstrate it as screwy even when the problem was otherwise active in furture "triggered" restarts. Though the same test scene does still result in 1/2 or more dying elsewhere on their road to the target, at least that part is repeatable between at least 5 "triggered" restarts so far where I've retested with same\very similar results.
From poking around it's as if the figure that's being fed to that unit-agl side of the equation wasn't being updated(or the update to it was being flagged to be bypassed and a stale value used, even then though I couldn't account for some of the actual negative values in there. What ever was going on ultimately the value compared was not correct for the state of the unit at the time, most of the time not even close.
If you're wondering about the popup it's just a weapon tracker system, where every second it tracks all the weapons I'm interested in and keeping running track of the last 15 seconds of locations. I tried sending some over the exact final steps of someone who crashed on land and it didn't replicate a crash on exactly the same path at the same time reliably so it didn't look 'exact' lat|lon specific. Interestingly doing the same tests away from the Kola, over in the Gulf of Mexico zig zagging out to depth and then to coastal then inland to targets around the contiguous states did NOT produce the same results when the problem was active. All those launches went without issue, same over in the Hawaii area, same over canada\AK\fareast Russia, or launches from cali to midwest over mountains. Didn't matter if it was 30 weapons active or 800 weapons active. I used several hundreds of missiles at a time traveling thousands of nm to spot check and cover a lot of
ground, nothing - except for in 40-90|20-60 and _only_ when the problem is "active" for me. It's not restricted to just the kola area, got several hits south of it as well, and some east of it. Makes me wonder if this does have an element of "e20n90.bgd" to it somehow some way.
On that front, shortly after starting op's scene sometimes I would get these in a pairs in once instance like 12 in another only 6.
1/28/2021 10:29:15 PM -- B1147.16 -- Index was outside the bounds of the array.
Exception: Index was outside the bounds of the array.
Stack Trace: at .... (Int16 , Int16 )
Call Stack & Error details:
Error at 200494, Index was outside the bounds of the array.
1/28/2021 10:29:15 PM -- B1147.16 -- Index was outside the bounds of the array.
Exception: Index was outside the bounds of the array.
Stack Trace: at .... (Int16 , Int16 , Int16 )
Call Stack & Error details:
Error at 200496, Index was outside the bounds of the array.
By sometimes I mean only on initial|first startup of the game where the problem was "active|triggered".
Those relate to lat|lon elevation cache array requests\lookup-and-fills I think. As for data that was fed to those two functions the first I couldn't tell you but the later one one occasion where I caught it was trying to index what I think is effectively a cache[latrow][loncol] it was trying to feed 5999|4445 to an array that only had the size of 7200x3600 or 3601 at the time. Almost like the request was based on one tile, but the cache being accessed was from another?? idk i couldn't figure out how it happened, the unit I think involved was a sat. I got those on two different startup per my logs, but soon as I wanted to dig into them more they never happened again on successive restarts where the overall issue was active, at least not yet.
As already alluded too and is probably obvious by now it's not restricted to 158A|B|C's I was able to get some 154C-1's (though it happened far less ~6-7 out of 384 fired in one long series) and 109E|I's to generate it too as well as kh-101, and as-22's, pretty sure it can effect at least anything that terrain follows in some fashion, 109Z-FastHawks for example didn't have the problem since they cruise at like 100k. I tested some aircraft flying at min alt with and without TG on, no crashes,on the startups when problem was 'active'. 15x didn't seem to make it happen more often for me. I got just as many on avg overall in 1x\2x as I did in 5\x15x, though I didn't keep a recorded table I'd actually say I got few more in 1x 2x 5x than 15x. I think that could just be a product of timing such 'where' the elevation check happens to be done. Though eventually if you keep your missiles in the air (refueling them with lua for testing purposes and moving them around, particularly to the right spots) they usually _all_ die in the end when problem is active, just a matter of where and when, just don't move them so far that they are technically out of range or they'll peter out. Oh and sea launched or air launched makes no difference when it's active it seems.
Now why does it appear to not always happen?
I don't know. FWIW I probably have one of the slower rigs out there winSrv2008r2 x64 us-eng i950 24gb and even at 2x,5x,15x,flame, I couldn't "trigger" it to happen on most startups. The few times it did show itself was after fresh restart (and creating new scene button to then load scene). This matches with my own experiences with this issue in the past, seemingly random and determined at startup, and cleared by a restart. If I was to speculate it almost seems like a product of how things end up scheduled during startup somehow (thread wise?)that can screw up perhaps the assumed order of events, and less about raw cpu capacity. Or maybe something to do with any pre-caching|setup of elevation data? Running it with only 1,2,3, cores made no difference in trying to repo it that I could find, running in low priority made no difference,suspending and un-suspending main thread didn't produce it or clear it up (not that anyone should be doing these things lol).
I'm stumped on how to purposely cause it to get into the state, total luck of the draw on startup. Once it's there it's fairly easy to repo but only inside one geographic area in my experiences. Now that doesn't mean it doesn't happen with other areas,as I only tested a few of the 33 regions during the few triggered starts I managed to get out of probably 100 restarts over a couple days. I did confirm there is nothing screwy with the SRTM30Plus data being pulled itself, the values returned when there is no exception are valid and appear correct for the lat|lon rows\cols being inserted into the cache when records were not yet cached, so it's not like there are dead spots or something in the data causing problems,at least not in the file.
I doubt it will help but I have full and mini dumps from the process while the problem state existed snapped shortly after the op's scene starts if devs want it, doubt it's that useful though. I did not focus much on the additionally mentioned 'overshot target' issue up thread, though I did see some in testing it wasn't more than a few that I just chocked up to rng, as I did the occasional legit rng malfunction so long as it happened near the target, but most what I was doing didn't involving letting munitions get to their targets in the first place.
On occasions you'll get one of these I think if you have a dead missile still selected I think, no biggy but figured it was worth mentioning.
-- B1147.16 -- Object reference not set to an instance of an object.
Exception: Object reference not set to an instance of an object.
Stack Trace: at Command.UnitEMCONViewModel.()
at Command.UnitEMCON_WPF. (ActiveUnit theUnit)
at Command.RightColumnWPF. (Scenario , Side , Unit )
Call Stack & Error details:
Error at 999999,
Attached another scene that when the problem happens you'll lose about 50-75% before the target depending on timing\jitter, but the path they crash on is pretty darn consistent overall. Though the op's is better for knowing you have the problem in the first minute or two, this one can take till they hit the coastal mountain region and the 19.9x to ~20.x' cross over, also it has far less going on to filter through if debugging and the units are customized for easier|further testing. It's setup to drop ref points on every destruction and message you but you
can turn that off in SA screen.
Some crash distribution screenshots:
[img]https://i.postimg.cc/4mz94RRc/Run1-2x-pattern1.jpg[/img]
[img]https://i.postimg.cc/sMmGSPXD/Run1-2x-pattern2.jpg[/img]
[img]https://i.postimg.cc/34sk7nL9/Run1-2x-pattern3.jpg[/img]
[img]https://i.postimg.cc/cgnJDfs3/Run1-KH-pattern12x.jpg[/img]
[img]https://i.postimg.cc/svcg0jRT/Run1-KH-pattern2-5x.jpg[/img]
Question for anyone who's had this happen:
When this happens to you has it ever happened outside 40->90N & 20->60E?
Does OP or others have this happen every time they start the game or is it like my system where it's a dice-roll?
Additionally when it happens is anyone else seeing those array bounds messages in the log (or anything else in the log) or were those just a me thing (and not even consistently a me thing at that)?
|
|
|
|
