I’ve noticed, when building terrains at very high resolutions, that WM can take a huge amount of memory, when many heightfields are involved at once.
I guess this is how WM manages heightfields when working with several devices:
it starts with the first device, which may require from one to several heightfields to be calculated
once the first device is calculated, the result can be one or several heightfields which get stored in memory
then on with the second device, which may have multiple outputs too, which will all be calculated and stored in memory
and so on with third device, fourth, etc etc…
This storing has a very useful purpose, which is the one of being able to change some parameters and re-build the terrain starting just from the device that has been changed; as well as being able to select any decive along the chain and check it’s specific output.
But this also takes a LOT of memory because every single device has one to several heightfields stored in memory which can be many and at high resolutions.
So I was thinking: why not being able to set which devices should reatain their heightfields in memory and which ones not?
Let me explain.
Let’s say we have a chain composed as:
[Layout Generator]->[Advanced Perlin]->[Clamp]->[Erosion]->[Coast Erosion]->[Select Height]->[Simple Transform]->[Height Output]
Now let’s say I’m very happy with the layout as well as with the erosion effect, but I want to play with the coast erosion and then with the simple transform to smooth some areas of the terrain.
All I would need is a heightfield for the erosion output to start with, as I would make no use of the ones that precede that device. And yet I have them all stored in memory.
So, according to my idea, I could set all those devices before the erosion one to not retain their heightfields in memory once they’re no longer needed for calculations.
Of course I could not step back to check their outputs, but I’ve already decided that’s not a part I want to change anymore. I think this would save a LOT of memory and would also allow me to render terrains at very high resolutions without taking too much RAM: once the output of a device is no longer needed by the following one, all its heightfields simply get unloaded.
I know I can somehow already do that by baking heightfields into RAW files, but this is not be a very dynamic workflow.
Imagine: baking the intermediate heightfield, disabling the calculated devices and then setting a file input to replace them in the chain. But what if I need to change something before that point? I have to re-enable the devices, re-render that part of the chain, save the file again, re-disable the devices, re-load the RAW file again…quite annoying and time wasting, to me.
Also, with the method I’m suggesting you could choose what heightfields to retain everytime you re-build the terrain, so that you can chose what steps of the chain are useful to be checked separately and could eventually need modifications, every single time you build it.
Plus, this would reduce the session status’s size quite a lot, I think, as we could write down just the devices’ outputs we’ll really need the next time we work with that terrain.
The ability to selectively retain is a good one, that I will put on my feature request list.
I wanted to double check however that you have tried using the Memory Conservation option, which retains only the finished height fields – this will not give you any interactivity, but it will cut down drastically on the memory required to do large final builds.
Combining the feature described here with memory conservation (select and flag sets of devices to retain their output in memory conservation mode) would be a very nice feature for working on huge size worlds!
Yep yep, I do use that feature to save some RAM, but it’s still a resolution of 16384x16384 multiplied by all the heightfields involved in the rendering…
I have 12GB of RAM, but it’s not enough to handle that much for a long chain… ^^’
EDIT: wow, 56 posts and I’m already a “World Machine Guru”!? =o
I would’ve said it to require at least 150-200 posts! ^^
Memory conservation SHOULD use the minimum of RAM possible for the network given – what causes large amounts to be used is not the network length, but its width – in other words, how many devices worth of data needs to be held in memory at one time for processing to proceed down the chain.
IE, in this crude ASCII graph if the number represents the number of device outputs being calculated at once in each “Stage” of the network:
1 - 2 - 12 - 2 - 4 - 2 - 1
The choke point then is obviously the part that needs 12 packets at once.
Two last questions:
Are you using WM 2.3beta (which fixes a few memory related issues),?
Have you set your physical ram target to something appropriate in the build options – unless it absolutely needs them for the currently processing device, WM can page out currently un-needed data to disk to free up space for what it does need.
No, still using 2.2. I have some kind of idiosyncrasy for beta versions and in this particular case I’d like to be sure I can backstep to any previous project without compatibility/bug problems.
Anyway I dont have any memory issue, it seems I just like to build complicated chains. ^^’
The target RAM I’ve set is 70%, which means around 8,5GB over 12. Now, the system alone eats about 2,6GB…I should still have 1GB+ of free memory, during rendering, and I do indeed. During renderings the memory usage never goes above 10,5GB.
EDIT: I’ve also posted an idea in the “Road painter” topic and I’d like to know what you think about it… =)
Would it be possible along with this “RAM output store option”, to have on each device the option to write down on HDD a temp file, similar with the one we have when building tiled builds? Considering that SSDs are pretty fast nowadays, I think it might help.
I`m even going further and think if would be possible to have some sort of “bucket renderer” and read parts of the paged file only when needed would lead to almost infinite worlds? Is it possible?
I read again my own post and I think having such an option would be great. RAM is volatile and having the data stored on disk for each device, not only for the whole session, would be great.
Having the option to store data on disk for each device, in a separate file for each device, in a WM session will also give the option to continue the work later, in another session. I know we can preserve now the whole session on disk, and indeed this can be really useful. I`m sure having such an option will open a lot of possibilities. Not only for the amount of memory used in a scene, or for the amount of saved time, but also workflow wise. Maybe even use the same devices in different scenes altogether. Or maybe helping building huge tiled worlds without even using the overlapping feature (Blending Percentage) option in the “Tiled Build Options” panel.