RE: need to shrink DB

Matrix Games Forums

Forums Register Login Photo Gallery Member List Search Calendars FAQ

My Profile Inbox Address Book My Subscription My Forums Log Out

RE: need to shrink DB

View related threads: (in this forum | in all forums)

Logged in as: Guest

Users viewing this topic: none

Printable Version

All Forums >> [New Releases from Matrix Games] >> War in the Pacific: Admiral's Edition >> RE: need to shrink DB

Page: << < prev 49 50 [51] 52 53 next > >>

Message

<< Older Topic Newer Topic >>

RE: need to shrink DB - 6/15/2010 7:18:15 PM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: herwin

quote:

ORIGINAL: floydg

quote:

ORIGINAL: herwin

That's a standard problem with JDBC systems. You need to bug the developers to allow people to use alternative high-performance DBMSs.

Okay, I'm game - if it can improve performance, can you recommend something?

There are two problems--big databases are unavoidably slower, and the memory fragmentation you get in the JVM as you load a lot of data means you start thrashing. There was this FAA system I had some involvement with that ran into just that problem. I've asked the DBMS expert who sits two desks over for some suggestions.

For a starter, is the database normalised?

I believe the database is in 3NF.

The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.

Floyd

(in reply to herwin)

Post #: 1501

RE: WitPTracker AE Release 1.0 - 6/16/2010 6:25:41 AM

n01487477

Posts: 4779
Joined: 2/21/2006
Status: offline

quote:

ORIGINAL: n01487477

quote:

ORIGINAL: jcjordan
I minor thing I noticed on the alert tab was that in my game as Allies vs AI IJ that on the new month I get the msgs about a/c research being moved forward to x month....

Let me get this right ... you're seeing your opponents R&D on the first of the month ?

Oops ... I think I wrote this bit of code I forgot to filter for side ... found, done and fixed

_____________________________

-Damian-
EconDoc
TrackerAE
Tutes&Java

(in reply to n01487477)

Post #: 1502

RE: need to shrink DB - 6/16/2010 8:29:43 AM

herwin

Posts: 6059
Joined: 5/28/2004
From: Sunderland, UK
Status: offline

quote:

ORIGINAL: floydg

I believe the database is in 3NF.

The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.

Floyd

What's probably killing the user is memory fragmentation during the load. You might look at advanced memory management techniques. Unfortunately, Java (and C#) aren't really designed for user control of memory usage. Do a Google search on Java memory fragmentation, and you'll find a number of articles on the subject. My colleague may get back to me later today.

_____________________________

Harry Erwin
"For a number to make sense in the game, someone has to calibrate it and program code. There are too many significant numbers that behave non-linearly to expect that. It's just a game. Enjoy it." herwin@btinternet.com

(in reply to floydg)

Post #: 1503

RE: need to shrink DB - 6/16/2010 2:28:50 PM

n01487477

Posts: 4779
Joined: 2/21/2006
Status: offline

quote:

ORIGINAL: herwin

quote:

ORIGINAL: floydg

I believe the database is in 3NF.

The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.

Floyd

What's probably killing the user is memory fragmentation during the load. You might look at advanced memory management techniques. Unfortunately, Java (and C#) aren't really designed for user control of memory usage. Do a Google search on Java memory fragmentation, and you'll find a number of articles on the subject. My colleague may get back to me later today.

Just spent a little bit of time doing some stats on Tracker ... jconsole.exe (JDK) -heap dump-> Mem Analyser(eclipse). Gave me what I thought in the first place, the map is a memory hog ! This was with about 10 turns loaded, I'll compare with 100+

Attachment (1)

_____________________________

-Damian-
EconDoc
TrackerAE
Tutes&Java

(in reply to herwin)

Post #: 1504

RE: need to shrink DB - 6/16/2010 5:49:22 PM

Zebedee

Posts: 535
Joined: 8/30/2005
Status: offline

Sent an e-mail to the techsupport address. Just can't get 64-bit Win 7 on this new AMD rig to recognise 32-bit Java :(

(in reply to n01487477)

Post #: 1505

RE: need to shrink DB - 6/17/2010 2:16:28 AM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: n01487477

quote:

ORIGINAL: herwin

quote:

ORIGINAL: floydg

I believe the database is in 3NF.

The amount of data loaded is transient - once we get a "current" snapshot, the memory is freed, so in steady-state, the usage is relatively constant. The problem is building the current snapshot, which is the time killer.

Floyd

What's probably killing the user is memory fragmentation during the load. You might look at advanced memory management techniques. Unfortunately, Java (and C#) aren't really designed for user control of memory usage. Do a Google search on Java memory fragmentation, and you'll find a number of articles on the subject. My colleague may get back to me later today.

Just spent a little bit of time doing some stats on Tracker ... jconsole.exe (JDK) -heap dump-> Mem Analyser(eclipse). Gave me what I thought in the first place, the map is a memory hog ! This was with about 10 turns loaded, I'll compare with 100+

Next release will have the ability to hide the map. It is a huge image read in...

(in reply to n01487477)

Post #: 1506

RE: need to shrink DB - 6/17/2010 7:20:08 AM

herwin

Posts: 6059
Joined: 5/28/2004
From: Sunderland, UK
Status: offline

Yes, but the map is fixed size--it shouldn't be a big issue. You could consider using a proxy and only loading it as needed, but then you might run into out of memory problems later.

_____________________________

(in reply to n01487477)

Post #: 1507

Replays Question - 6/17/2010 9:34:57 AM

herwin

Posts: 6059
Joined: 5/28/2004
From: Sunderland, UK
Status: offline

I've offered my opponent a 'mulligan' back about 5 days. I assume I need to reload all the .pws files up to that date...

_____________________________

(in reply to herwin)

Post #: 1508

RE: Replays Question - 6/17/2010 11:42:59 PM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: herwin

I've offered my opponent a 'mulligan' back about 5 days. I assume I need to reload all the .pws files up to that date...

Yeah, no way to "unload" turns.

(in reply to herwin)

Post #: 1509

RE: need to shrink DB - 6/17/2010 11:44:25 PM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: herwin

Yes, but the map is fixed size--it shouldn't be a big issue. You could consider using a proxy and only loading it as needed, but then you might run into out of memory problems later.

The loading takes a huge amount of time in addition to huge amount of memory. I'd need to profile it a little deeper to see where the bottleneck is, but for now, as I said, the option to turn it off will be available.

(in reply to herwin)

Post #: 1510

RE: need to shrink DB - 6/18/2010 12:08:27 AM

witpqs

Posts: 26087
Joined: 10/4/2004
From: Argleton
Status: offline

Floyd, personally I've stopped using the map and just use the 'Z' key in-game. Having the search arcs on the map would be nice, but (reported a while back) they don't work right. Having the Z key, while not as good an overview as Tracker, is good enough. So an option to turn off the map would be great.

To recap, the problem with the search arcs is this (made up example to illustrate the key point):

Take a group with 9 planes and put it on 100% search. Tell it to search from 0d to 180d. Do 'show arcs' and you will see a morning search for half of the search (say 0d to 90d) and an afternoon search for the other half (say 90d to 180d).

Instead tell it to search 0d to 90d. Do 'show arcs' and you will see that 0d to 90d is searched both morning and afternoon.

Tracker - in both cases - will show a morning search for half and an afternoon search for half. In other words, it seems like Tracker just takes the starting place (0d), takes the number of planes set to search (planes and percentage set to search), and then counts out morning and afternoon searches. The ending place (90d) is not recognized.

(in reply to floydg)

Post #: 1511

RE: need to shrink DB - 6/18/2010 12:26:05 AM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: witpqs

Floyd, personally I've stopped using the map and just use the 'Z' key in-game. Having the search arcs on the map would be nice, but (reported a while back) they don't work right. Having the Z key, while not as good an overview as Tracker, is good enough. So an option to turn off the map would be great.

To recap, the problem with the search arcs is this (made up example to illustrate the key point):

Take a group with 9 planes and put it on 100% search. Tell it to search from 0d to 180d. Do 'show arcs' and you will see a morning search for half of the search (say 0d to 90d) and an afternoon search for the other half (say 90d to 180d).

Instead tell it to search 0d to 90d. Do 'show arcs' and you will see that 0d to 90d is searched both morning and afternoon.

Tracker - in both cases - will show a morning search for half and an afternoon search for half. In other words, it seems like Tracker just takes the starting place (0d), takes the number of planes set to search (planes and percentage set to search), and then counts out morning and afternoon searches. The ending place (90d) is not recognized.

First time I've heard about this bug. But I see where in the code I've gone wrong: not checking the end position and just blowing through it.

Floyd

(in reply to witpqs)

Post #: 1512

RE: need to shrink DB - 6/18/2010 9:17:02 PM

Zebedee

Posts: 535
Joined: 8/30/2005
Status: offline

Just wanted to thank Floyd and the WitPtracker team for helping me get up and running.

Nice one :)

(in reply to floydg)

Post #: 1513

Base data - 6/19/2010 5:58:34 PM

Roger Neilson II

Posts: 1517
Joined: 7/16/2006
From: Newcastle upon Tyne. England
Status: offline

Have just tried an export of the Bases material - very useful potentially, but whilst it has supply data there is no fuel data? Oe of the key things i was looking for?

Roger

_____________________________

(in reply to Zebedee)

Post #: 1514

RE: Base data - 6/19/2010 7:13:59 PM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: Roger Neilson II

Have just tried an export of the Bases material - very useful potentially, but whilst it has supply data there is no fuel data? Oe of the key things i was looking for?

Roger

The export function only exports what's shown in the table. I can add fuel to the table so next release you'll be able to carefully count your barrels.

Floyd

(in reply to Roger Neilson II)

Post #: 1515

RE: Base data - 6/19/2010 7:35:22 PM

Roger Neilson II

Posts: 1517
Joined: 7/16/2006
From: Newcastle upon Tyne. England
Status: offline

That would be really useful.

Roger

_____________________________

(in reply to floydg)

Post #: 1516

RE: need to shrink DB - 6/19/2010 9:07:42 PM

witpqs

Posts: 26087
Joined: 10/4/2004
From: Argleton
Status: offline

quote:

ORIGINAL: floydg

First time I've heard about this bug. But I see where in the code I've gone wrong: not checking the end position and just blowing through it.

Floyd

Thanks!

(in reply to floydg)

Post #: 1517

RE: need to shrink DB - 6/19/2010 11:17:22 PM

vonTirpitz

Posts: 511
Joined: 3/1/2005
From: Wilmington, NC
Status: offline

version 1.5.1

I have observed a minor nuisance that occurs when trying to load the Industry screen after some of the other screens have been viewed. For example, if I load and review the events, check out the victory screen and perhaps the Intel screen then go to Industry it won't display. I usually have to select another screen such as mines or whatnot and then the Industry screen again. It generally displays properly when selected after that. Kind of feels like perhaps a flag isn't being initialized set or cleared but just guessing.

Great utility though. Saves a player a lot of time trying to figure out what is where.... Thanks for all the hard work!

Marc

(in reply to witpqs)

Post #: 1518

RE: need to shrink DB - 6/20/2010 4:11:25 AM

erstad

Posts: 1944
Joined: 8/3/2004
From: Midwest USA
Status: offline

version 1.5.1

Very minor issue, but might be easy to address if it seems worthwhile.

On the LCU screen, when filtering for "Lack supply", all the LCU fragments loaded on ships show up, because the LCU drops its supply when it loads.

Since all loaded LCUs have no supply, and there's a lot of fragments, seems like it would be better to only list the LCUs on the ground.

(in reply to vonTirpitz)

Post #: 1519

RE: need to shrink DB - 6/21/2010 3:26:57 AM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: erstad

version 1.5.1

Very minor issue, but might be easy to address if it seems worthwhile.

On the LCU screen, when filtering for "Lack supply", all the LCU fragments loaded on ships show up, because the LCU drops its supply when it loads.

Since all loaded LCUs have no supply, and there's a lot of fragments, seems like it would be better to only list the LCUs on the ground.

Fixed.

(in reply to erstad)

Post #: 1520

RE: need to shrink DB - 6/21/2010 3:35:39 AM

n01487477

Posts: 4779
Joined: 2/21/2006
Status: offline

Updated the ShipProduction screen ... making it easier to read (Hopefully)

Attachment (1)

_____________________________

-Damian-
EconDoc
TrackerAE
Tutes&Java

(in reply to floydg)

Post #: 1521

Out with the old. In with the new... - 6/21/2010 6:11:33 PM

n01487477

Posts: 4779
Joined: 2/21/2006
Status: offline

Thought I'd update the LCU Production screen at the same time ... now allows full screen too ... damn I hate Java GUI.

Attachment (1)

_____________________________

-Damian-
EconDoc
TrackerAE
Tutes&Java

(in reply to n01487477)

Post #: 1522

WIN 7 64 bit system - 6/24/2010 7:43:13 AM

davidjruss

Posts: 235
Joined: 5/25/2002
From: Derby, England
Status: offline

I have tracker working fine on a 32bit XP system with Java but was wondering if there sre any full installation instructions available anywhere for getting tracker to work on a WIN 7 64 bit AMD system laptop with the latest JAVA installed ( e.g get an error message stating cannot load 32 bit dll on a 64 bit AMD system ) Trying to read through 51 pages of the tracker thread for info on WIN 7 is making my eyes hurt.

Davidr

(in reply to n01487477)

Post #: 1523

RE: WIN 7 64 bit system - 6/24/2010 7:52:57 AM

n01487477

Posts: 4779
Joined: 2/21/2006
Status: offline

quote:

ORIGINAL: DavidR

I have tracker working fine on a 32bit XP system with Java but was wondering if there sre any full installation instructions available anywhere for getting tracker to work on a WIN 7 64 bit AMD system laptop with the latest JAVA installed ( e.g get an error message stating cannot load 32 bit dll on a 64 bit AMD system ) Trying to read through 51 pages of the tracker thread for info on WIN 7 is making my eyes hurt.

Davidr

Error Msg:
1. Can't load IA 32-bit .dll on a AMD 64-bit platform
Vista 64 and Intel / AMD
This error message is usually because you haven�t installed the Microsoft C++ libraries. But if you have then it can also be because you�ve not got Java 32 installed and configured properly.

Solutions.
Step 1. Check C++ Libraries.
Check you have the C++ Libraries installed, if not, install the 32 bit version (even if you have a 64 bit OS.)

# I'm unsure if you can have both the 32 and 64 bit C++ libraries available and still get Tracker to run, or how to configure it ... so it�s best for now just to have 32 bit for either 32 or 64 bit OS.

Check here for installation:
http://sites.google.com/site/witptracker/installation

Step 2. Check you have 32 bit Java
Check you have 32 bit Java installed. If you are working on a 64 bit OS, then you can have both installed.

Step 2. Option 1. Change your witptracker.bat file to point to the 32 bit Java executable (you need the ""'s cause of the space) - this varies on different machines so you'll need to check the path:
32bitOS users
C:\"Program Files"\Java\jre6\bin\java -Xms384m -Xmx512m -jar "WitPTracker.jar"
pause
64bitOS users
"C:\Program Files (x86)\Java\jre6\bin\java.exe" -Xms384m -Xmx512m -jar "WitPTracker.jar"
pause

NB** you must run WitpTracker from the batch file !!
(See link for executing the batch file in Win64 bit if it won�t run http://sites.google.com/site/witptracker/support )
OR
there is an executable Tracker file that might solve your problem IF you've done the step above http://sites.google.com/site/witptracker/support (end of the page)

_____________________________

-Damian-
EconDoc
TrackerAE
Tutes&Java

(in reply to davidjruss)

Post #: 1524

RE: WIN 7 64 bit system - 6/24/2010 8:22:23 AM

davidjruss

Posts: 235
Joined: 5/25/2002
From: Derby, England
Status: offline

Damian,

Thank you for the info.

However I seem to have 2 folders with Java on my WIN 7 64 bit system ( 1 JAVA folder in program files and 1 JAVA folder in program files (x86) ) . The JAVA folder in program files appears to be the original and has not been updated. The JAVA folder in program files (x86) has been updated.

Not wanting to break my OS and not being very computer literate is it OK to just delete the JAVA folder in the (x86) program files folder and then install JAVA 32 bit ( if I can find such a beast , as my system updates JAVA automatically for me without any manual input ) ?

Davidr

(in reply to n01487477)

Post #: 1525

RE: WIN 7 64 bit system - 6/24/2010 8:29:40 AM

herwin

Posts: 6059
Joined: 5/28/2004
From: Sunderland, UK
Status: offline

I have both versions of Java installed. The bat file I use is:

C:\"Program Files (x86)"\Java\jre6\bin\java -Xms384m -Xmx512m -jar "WitPTracker.jar"
pause

_____________________________

(in reply to davidjruss)

Post #: 1526

RE: WIN 7 64 bit system - 6/24/2010 8:32:38 AM

n01487477

Posts: 4779
Joined: 2/21/2006
Status: offline

quote:

ORIGINAL: DavidR

Damian,

Thank you for the info.

However I seem to have 2 folders with Java on my WIN 7 64 bit system ( 1 JAVA folder in program files and 1 JAVA folder in program files (x86) ) . The JAVA folder in program files appears to be the original and has not been updated. The JAVA folder in program files (x86) has been updated.

Not wanting to break my OS and not being very computer literate is it OK to just delete the JAVA folder in the (x86) program files folder and then install JAVA 32 bit ( if I can find such a beast , as my system updates JAVA automatically for me without any manual input ) ?

Davidr

Definitely DO NOT delete any files ... go and download the 32 bit Java http://www.java.com/en/download/manual.jsp & install if. Maybe at present you have the 32 bit installed as well... but then you'll need to change the batch file to that (x86) path and hopefully it'll be working.
Also check you have 32bit C++ libraries ... Control Panel -> add / remove programs.

Good luck

[edit:] Herwin's path should work ...

< Message edited by n01487477 -- 6/24/2010 8:33:16 AM >

_____________________________

-Damian-
EconDoc
TrackerAE
Tutes&Java

(in reply to davidjruss)

Post #: 1527

Feature request - 6/24/2010 2:22:12 PM

Lifer

Posts: 384
Joined: 6/8/2003
From: Caprica
Status: offline

Playing wide screen and full screen. Can Tracker be made to stay on top to allow scrolling around on the map to areas of interest that I find in tracker? I don't know if this would help anyone else but I would find it useful.

Greg

_____________________________

Man does not enter battle to fight, but for victory. He does everything that he can to avoid the first and obtain the second.
Ardant du Picq

(in reply to n01487477)

Post #: 1528

RE: WIN 7 64 bit system - 6/24/2010 3:00:06 PM

davidjruss

Posts: 235
Joined: 5/25/2002
From: Derby, England
Status: offline

Damian and Herwin,

Many thanks for your advice , I have now got Tracker working on my WIn 7 64 bit system.

Davidr

(in reply to n01487477)

Post #: 1529

RE: Feature request - 6/24/2010 7:18:43 PM

floydg

Posts: 2052
Joined: 6/27/2004
From: Middletown, NJ
Status: offline

quote:

ORIGINAL: Lifer

Playing wide screen and full screen.� Can Tracker be made to stay on top to allow scrolling around on the map to areas of interest that I find in tracker?� I don't know if this would help anyone else but I would find it useful.

Greg

It's possible. Let me see if I can wrangle that in.

EDIT: it's possible. I added a menu item to toggle the "keep on top" state. Appears to work fine.

Floyd

< Message edited by floydg -- 6/26/2010 1:59:20 PM >

(in reply to Lifer)

Post #: 1530