World Machine Community

General => General Discussion => Topic started by: jgwinner on February 27, 2005, 09:58:27 pm

Title: Feature Request - partially complete recalculation
Post by: jgwinner on February 27, 2005, 09:58:27 pm
I would REALLY like a way to be able to take a very large (8k x 8k) height field and start the calculation, stop it, exit out, then restart where the calculation left off.

I realize this could be hard, but sometimes things take a LONG time to complete.  

So ... as a step toward this, a 'pause' button would be VERY nice.  Sometimes you need to answer an email and you don't want to spoil an hours long run ...

(this may have already been mentioned, if so I'm just adding another vote  :) )


     == John ==
Title: Feature Request - partially complete recalculation
Post by: Fil on February 28, 2005, 03:27:42 am
Well, some time back in the preview of feature WM v1.0 would have, I believe it was mentioned in a way sligly different.
v1.0 will have a completely rewritten memory management.. That includes trashing or keeping each device's build result. This means that the next time you build the world, some devices won't be built again (like it happens now) The difference is that some of the outputs of certain devices may desabpear. However, this will be a user-controlled option, somewhere in the settings.
Anyway, currently you can stop a build and continue afterwards. But this is limited to two things:
it will stop at a granularity of a device (meaning you if you stop the build at the middle of a device, that info might be lost)
it won't be saved between sessions (if you close WM, it won't restart from where it finished)
So these are the only two points that could be addressed.

Saving heightmaps to disk when exiting WM, could solve the second issue, with the expense of possibly huge .tmd files.
Solving the first issue would be trickier, as each device had to be reworked so that it could store temporary data and take up from temporary data.. Some algorithms might not be very easy to tweak this way.. The most time consuming of them, being the erosion one. When a device became "paused", it had to write its entire state to disk, and be able to recover from there..

On the other hand, the largest size we can actually work on is 4Kx4K.. 8Kx8K would involve 4x more memory needed. I know v1.0 will have improved memory management, thus ocupying less memory, however, I get arround 60MB occupied with 512x512 in v.99 I imagine 8Kx8K (256 times larger) would reach undesireable values (perhaps larger than 1GB) I know there is swap, but.. it would sound like overkill.

Instead the best way to increase terrain size ( may be to do smaller tiles and join them together (, with the associated difficulties of stitching ( and/or using a small set of scalable/stabilized devices.
Title: Feature Request - partially complete recalculation
Post by: Stephen on February 28, 2005, 06:33:45 pm
The pause button is a pretty good idea, and doable -- WM hogs the processor pretty badly while building, so it would be nice to be able to stop it for a moment and then continue.

The first suggestion would be much harder, due to the sheer volume of data that would have to be stored for this to work.. Realize, at extreme terrain sizes (8192), a heightfield takes up 256MB each. So to just do one operation -- erosion, would require almost a 1gb of space.... Saving that out to disk would be pretty size prohibitive, not to mention how your RAM is doing :)
Title: Feature Request - partially complete recalculation
Post by: Anonymous on March 01, 2005, 08:36:55 am
>>I imagine 8Kx8K (256 times larger) would reach undesireable values (perhaps larger than 1GB)<<

And your point is?  :D

My machine currently has 1.5G and the only reason I don't go to 2G is that I'm running XP currently.   Even my wife's PC has 1G.

Even so, it's not a bad thing to go into swap, especially if it's an hours long calcuation.

It sounds like spilling intermediate calculations to disk are too hard, but if I could (through a setup option) set the sizes larger and let VM take it, then I could do a (days) long calculation and still do what I need.

Of course, the idea of tiling would be more practical and allow even large terrain to be constructed.  Saving the edge conditions would be hard, but maybe a special file format for this.

Thinking through the use case for this - I usually start with a large, low rez file (MOLA data from Mars) and then 'cut it up' into squares.  If you did it this way (or  a perlin initial 'master file'), the edge info and averages could be derived from the master file.  Then all you'd have to do is save the actual high rez edges in a strip.  Maybe a special format file for this, that would be basically 1 or 2 pixes from the top, left, right, bottom edges; the file would be the size of the grid.  i.e. if you had a tile grid of say 5 tiles the edge guide grid would be 6x6 pixels.  If erosion need more data to get the slopes right at the edge, then it would be n+1 in size.

Will non rectangular files be possible?  that would change the above algorithm to (m+1)x(n+1) (and would be nice  :) )

    == John ==