Matrix Games Forums

Forums  Register  Login  Photo Gallery  Member List  Search  Calendars  FAQ 

My Profile  Inbox  Address Book  My Subscription  My Forums  Log Out

An auto-mapper for TOAW

 
View related threads: (in this forum | in all forums)

Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [New Releases from Matrix Games] >> The Operational Art of War IV >> Mods and Scenarios >> An auto-mapper for TOAW Page: [1]
Login
Message << Older Topic   Newer Topic >>
An auto-mapper for TOAW - 12/16/2021 12:15:05 PM   
CaptainMaxwell

 

Posts: 13
Joined: 7/13/2013
From: Vienna
Status: offline
Well, kind of.

Fiddling around with Armoured Brigade last year. They have some documentation on how to extract base map data from DEM maps and OSM. After playing around in QGIS I realized that one can do so much more.

In about three months of work I developed a process in QGIS that can take data from a DEM (like radar height map) and a vector base map (think Google Maps) and construct a TOAW map out of it. The results are simply astonishing: I was able to draw a 75,000 hex map with forests, mountains, escarpments, rivers, roads and annotations in an afternoon. Accurate and pixel perfect (for better or worse).

This is a screenshot of part of the basemap in QGIS (If you are curious, this is, in austrian military parlance, the ,,Bruck gate''. This is the path that the Red Army used in '45 and that supposedly the SGF would use in WWIII).


This is a screenshot of the part of the computed TOAW map in QGIS.


If you download them and switch back and forth between them in your image viewer application, you can compare terrain features and labels. You can also see the activated layers for the TOAW map. It's basically a geo-referenced table of a bazillion hexagons in QGIS that collect the data for the respective tiles during the creation process and are then textured in QGIS using the in-game textures. I wrote an exporter function in python that can then produce a mml-file out of this mess.

This is a screenshot of the part of the loaded map in TOAW.


I have also a dump of the complete map from TOAWs editor, but that file is simply too large.

The basic process is straight forward enough (if you have the base data), but coming up with (TOAW-specific) solutions to auto-build the proper contours, to make sure that both types of roads and rivers work together and that escarpments are created automatically gave me some serious headaches. I also tried to create rocky terrain, but you would need laser scan data to do that (normal DEM data is too coarse). Terrain features that you can not find in Europe are not implemented (but the process can be adapted).

This is of course of no use for historical maps, but for contemporary ones where there is data this can speed up processes by orders of magnitude.

Mapping and wargaming was on the backseat for about a year now, and only recently did I return to it. I saw r williams efforts over at the other thread, and thought that he could make use of it. There is enough DEM data available for Europe, and for 1990 free CLC vector data is also available (actually, for most of Europe).

You can find the process manual as attachment. It should give a complete walkthrough for dummies. I will keep the style file, texture sets, and the coverage and export scripts back unless someone really needs them.

I should also mention the associated caveats:
1.) The screenshots show the maximum solution. This is built from an official government dataset. Those usually cost a fortune, and open alternatives generally neither have the quality nor the detail. But then you usually don't want to go all the way to even importing river names and such.
2.) Even though the process builds a map within hours, you will most likely spend weeks learning the basics of QGIS and applying it to the base data. You have to consolidate, clean and repair it (yes, even for government grade data), understand the data structure, and select and filter the features out such that the process can eat them. If you feed it with trash, it will produce trash.
3.) This is a global, robotic process. Save for the water calculation I employed no fuzzification. The process will steamroll over the whole map as per global settings. The terrain calculation is not fixed (you will have to provide values to it; except for the escarpment detection, but those can be easily changed), but the coverage calculation for the moment is. But I am happy to discuss alternative values; currently forestation seems to be a little bit to common.
4.) This is pixel perfect, without abstraction. This is fair game for terrain and coverage, but should be treated with care especially for the roads as you will end up with bends and kinks more often than you like. Or roads through water.
5.) The way I apply the contouring and overlapping coverage textures may not be universally accepted, but I like the map to look ,,good'' (for whatever that means). Unless it breaks something in game I have no intent to rebuild that.

And since I was asked: I hope that I can migrate the process to Modern Campaigns, using the above mentioned CLC data. The process can be applied to everything, as long as I know (or can reverse engineer) the map structure. And it can't be more complicated than TOAWs contouring...

Attachment (1)

< Message edited by CaptainMaxwell -- 1/9/2022 8:03:21 AM >


_____________________________

Maxwell is always right.
He may be misinformed,
inexact, bullheaded, fickle,
ignorant, even abnormally
stupid, but
NEVER WRONG!
Post #: 1
RE: An auto-mapper for TOAW - 12/16/2021 12:27:36 PM   
Lobster


Posts: 5104
Joined: 8/8/2013
From: Third rock from the Sun.
Status: offline
I think you have to make two or three more posts before the forum will allow you to post things like pictures or links.

_____________________________

http://www.operationbarbarossa.net/

Two things are infinite: the universe and human stupidity and I’m not sure about the universe-Einstein

Q: What do you call a boomerang that doesn’t come back?
A: A stick.

(in reply to CaptainMaxwell)
Post #: 2
RE: An auto-mapper for TOAW - 12/16/2021 1:00:19 PM   
LOK32MK

 

Posts: 45
Joined: 7/19/2015
Status: offline
I am very interested! I used Google earth, hexthingy and a bunch of GIS files (DEMs, ground cover, population stats etc.) with Matlab scripts to do what you described. It took me several weeks and it was rather painful. I would love to see your work. Even for historical maps it would be extremely useful. The borders may have changed but the topography hasn't changed that much.

(in reply to Lobster)
Post #: 3
RE: An auto-mapper for TOAW - 12/16/2021 6:09:33 PM   
CaptainMaxwell

 

Posts: 13
Joined: 7/13/2013
From: Vienna
Status: offline
Nothing to see here. Everything incorporated into the IP.

< Message edited by CaptainMaxwell -- 12/28/2021 10:14:24 AM >

(in reply to LOK32MK)
Post #: 4
RE: An auto-mapper for TOAW - 12/16/2021 8:11:57 PM   
sPzAbt653


Posts: 9511
Joined: 5/3/2007
From: east coast, usa
Status: offline
I generally use Google Maps and HexThingy to make a TOAW map, and then make adjustments based on historical references. So maybe your work can be used for more than contemporary maps.

(in reply to CaptainMaxwell)
Post #: 5
RE: An auto-mapper for TOAW - 12/17/2021 12:00:15 PM   
CaptainMaxwell

 

Posts: 13
Joined: 7/13/2013
From: Vienna
Status: offline
Nothing to see here. Everything incorporated into the IP.

< Message edited by CaptainMaxwell -- 12/28/2021 10:14:13 AM >

(in reply to sPzAbt653)
Post #: 6
RE: An auto-mapper for TOAW - 12/17/2021 1:49:03 PM   
golden delicious


Posts: 5575
Joined: 9/5/2000
From: London, Surrey, United Kingdom
Status: offline

quote:

ORIGINAL: CaptainMaxwell

So... in principle I can now bump this, get my tenth post, and should be able to modify the first post to include the proper things. Let's see if this diabolical scheme works.

Edit:
Not quite, but almost. Attached is a zip with the three relevant images. Everything else should now be part of the original post.

No. 1 is a screenshot of part of the basemap in QGIS (If you are curious, this is, in austrian military parlance, the ,,Bruck gate''. This is the path that the Red Army used in '45 and that supposedly the SGF would use in WWIII).

No. 2 is a screenshot of the part of the computed TOAW map in QGIS.

No. 3 is a screenshot of the part of the loaded map in TOAW.

I have also a dump of the complete map from TOAWs editor, but that file is simply to large.


This looks good. Am I right in saying that the program is actually looking at how rugged the terrain is, rather than simply using elevation to determine what's a hill? If so that's exceptional.

What I (and anyone designing historic scenarios) would want, would be an option to omit most of the human geography; not to place any urban, dense urban, road, improved road, rail, airfield, farmland or bocage hexes. Of course the designer then has to put all these things back in, but I think that would lead to better results than cutting down from the modern picture. I can cope with any erroneous reservoirs etc.

_____________________________

"What did you read at university?"
"War Studies"
"War? Huh. What is it good for?"
"Absolutely nothing."

(in reply to CaptainMaxwell)
Post #: 7
RE: An auto-mapper for TOAW - 12/17/2021 5:36:47 PM   
CaptainMaxwell

 

Posts: 13
Joined: 7/13/2013
From: Vienna
Status: offline
quote:

This looks good. Am I right in saying that the program is actually looking at how rugged the terrain is, rather than simply using elevation to determine what's a hill? If so that's exceptional.


I am using the median of the slope (1st derivative). IIRC using ruggedness (2nd derivative) gave inferior results. The height (or rather height difference between neighboring hexes) is only used for placing escarpments.

quote:

What I (and anyone designing historic scenarios) would want, would be an option to omit most of the human geography; not to place any urban, dense urban, road, improved road, rail, airfield, farmland or bocage hexes. Of course the designer then has to put all these things back in, but I think that would lead to better results than cutting down from the modern picture. I can cope with any erroneous reservoirs etc.


The process is modular. If you just want terrain and water you are good to go. Just leave out what you don't want.

If you want to have forests only (because that's the cover type you did not explicitly mention) this is doable as well - the manual gives some advice on this --- but without some filler polygons results will probably be suboptimal.

_____________________________

Maxwell is always right.
He may be misinformed,
inexact, bullheaded, fickle,
ignorant, even abnormally
stupid, but
NEVER WRONG!

(in reply to golden delicious)
Post #: 8
RE: An auto-mapper for TOAW - 12/17/2021 5:54:38 PM   
golden delicious


Posts: 5575
Joined: 9/5/2000
From: London, Surrey, United Kingdom
Status: offline

quote:

ORIGINAL: CaptainMaxwell

I am using the median of the slope (1st derivative). IIRC using ruggedness (2nd derivative) gave inferior results. The height (or rather height difference between neighboring hexes) is only used for placing escarpments.


Perfect.

_____________________________

"What did you read at university?"
"War Studies"
"War? Huh. What is it good for?"
"Absolutely nothing."

(in reply to CaptainMaxwell)
Post #: 9
RE: An auto-mapper for TOAW - 12/17/2021 6:04:10 PM   
ncc1701e


Posts: 7380
Joined: 10/29/2013
From: Utopia Planitia Fleet Yards
Status: offline
Nice!

_____________________________

Chancellor Gorkon to Captain James T. Kirk:
You don't trust me, do you? I don't blame you. If there is to be a brave new world, our generation is going to have the hardest time living in it.

(in reply to golden delicious)
Post #: 10
RE: An auto-mapper for TOAW - 12/17/2021 10:48:36 PM   
ernieschwitz

 

Posts: 3893
Joined: 9/15/2009
From: Denmark
Status: offline
Sounds like you did a lot of work. I am pleased that it is possible, though I understand that there are some caveats. But it does sound pretty awesome. Of course this does take some of the artistic creativity away... which is something I think many can live with, but you also mentioned that you need to look it over and clean it up, so I imagine that is where some of the artistry will re-enter.

_____________________________

Creator of High Quality Scenarios for:
    Advanced Tactics Gold
    DC: Warsaw to Paris
    DC: Community Project.

Try this Global WW2 Scenario: GD1938v3

(in reply to ncc1701e)
Post #: 11
RE: An auto-mapper for TOAW - 12/17/2021 11:41:40 PM   
Simon Edmonds

 

Posts: 132
Joined: 5/10/2018
Status: offline
Nice Work. How good do you feel it is at picking up escarpments?

(in reply to ernieschwitz)
Post #: 12
RE: An auto-mapper for TOAW - 12/18/2021 3:35:39 PM   
cathar1244

 

Posts: 1003
Joined: 9/5/2009
Status: offline
Wow, excellent work done here.

I've messed a bit with QGIS, but this looks really complex. For example, as a beginner, I would have no idea how to ensure the vector data was comparable with the terrain data set. It sounds like it could become a pay-to-use service for those who aren't masters of QGIS and digital mapping.

Dunno. I would be happy with an editor that allows me to lay a hex overlay on any kind of map image and then allow me to assign TOAW terrain tiles to the hexes, with the result being saved as a .mml file.

Cheers

(in reply to Simon Edmonds)
Post #: 13
RE: An auto-mapper for TOAW - 12/19/2021 9:54:44 AM   
CaptainMaxwell

 

Posts: 13
Joined: 7/13/2013
From: Vienna
Status: offline
quote:

ORIGINAL: Simon Edmonds
Nice Work. How good do you feel it is at picking up escarpments?


A solid Meh out of 10. This seems to work in the alpine environment in the west of the country where valleys limit you now to the central hex unless you have some serious climbing gear. There is a benchmark feature (a fault scarp) in the plains north of Vienna that it cannot find, however. At least not unless I change the detection variables to ridiculous values and the whole map gets plastered with escarpments.

But this is most likely due to limits in the DEM. Generally available DEMs have a resolution of 90m or 30m. I am using a 10m set already. The larger ones will probably need serious manual corrections. I was thinking using rescaling and deconvolving to improve this, but real world image manipulation is not an episode of CSI. Besides, I would need to use workplace resources for that.

quote:

ORIGINAL: cathar1244
For example, as a beginner, I would have no idea how to ensure the vector data was comparable with the terrain data set.


If your base data is properly geo-referenced this should not be a problem for the user. Granted, I was pulling DEM and vector base data from the same source, so this was guaranteed; this is not necessarily the case if you grab the data from some web server. But reprojection should be only a mouse click. With that said, QGIS 3.16 did not convert the created slope maps properly (3.15 did); but I suppose that this was rectified in the year since then (its now 3.22).

You can have a look at the Copernicus and Corine data from the EU (this covers most of Europe). This should work together and produce proper height and and vegetation features in TOAW. I think it's a good solution for what golden delicious was asking for. Resolution is limited ofc, but TOAW maps only rarely go below 10km/hex anyway.

_____________________________

Maxwell is always right.
He may be misinformed,
inexact, bullheaded, fickle,
ignorant, even abnormally
stupid, but
NEVER WRONG!

(in reply to Simon Edmonds)
Post #: 14
RE: An auto-mapper for TOAW - 12/19/2021 11:52:36 AM   
cathar1244

 

Posts: 1003
Joined: 9/5/2009
Status: offline
I'm sure you're correct.

I am just put off by QGIS. As I'm not familiar with how digital mapping works, it was not obvious how to do basic operations, and it seemed most of the time that QGIS either complained or produced something that I wasn't looking for. And the documentation didn't help most of the time.

I suspect I would have to spend a day or two with you going through the operations to get comfortable with them. Ah well.

Looking forward to see what kind of maps you make with the tool.

Cheers

(in reply to CaptainMaxwell)
Post #: 15
RE: An auto-mapper for TOAW - 12/22/2021 7:55:42 PM   
MausMan2


Posts: 65
Joined: 8/13/2007
From: Minnesota, USA
Status: offline
Thanks CaptainMaxwell, As and esri gis user -and a QGIS user for fun on the side, I often though I could be ambitious enough to come up with a Python script that could fully automate mapping for the game. -didn't happen, but glad you did it. I've been using OnTopReplica, but certainly want to start using what you've developed.

(in reply to cathar1244)
Post #: 16
RE: An auto-mapper for TOAW - 12/28/2021 10:13:56 AM   
CaptainMaxwell

 

Posts: 13
Joined: 7/13/2013
From: Vienna
Status: offline
I finally edited the first post to include all the imagery. Hooray.

quote:

ORIGINAL: cathar1244
It sounds like it could become a pay-to-use service for those who aren't masters of QGIS and digital mapping.


No, thanks. With that said, I am happy to provide support to anyone who can provide the base data. Getting the (vector) data is the real problem. If you have that, you can simply follow the manual.

_____________________________

Maxwell is always right.
He may be misinformed,
inexact, bullheaded, fickle,
ignorant, even abnormally
stupid, but
NEVER WRONG!

(in reply to cathar1244)
Post #: 17
RE: An auto-mapper for TOAW - 1/9/2022 5:26:50 AM   
RyanCrierie


Posts: 1461
Joined: 10/14/2005
Status: offline
Photos aren't loading.

_____________________________


(in reply to CaptainMaxwell)
Post #: 18
Page:   [1]
All Forums >> [New Releases from Matrix Games] >> The Operational Art of War IV >> Mods and Scenarios >> An auto-mapper for TOAW Page: [1]
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

0.891