Solutions for building high-res worlds WITHOUT tiling?

[JackIt999]

I am trying to build a rather massive 50,000x30,000 resolution world and I have reached the point where I need to use Flow Restructure. Given that this device must account for the terrain as a whole, I cannot perform a tiled build. I have tried to perform a single build at this resolution, but I come to a fatal memory allocation failure. I am not sure if this is a bug, or if my PC simply does not have enough space for disk swapping to occur. (I currently have 400GB available for paging)

If this is NOT a bug, would it be advisable to purchase an external hard drive with more space to use for paging? If so, would a SSD or HDD be better? I suspect I would need multiple terabytes of space at this point.

My specs:

WM Professional edition build 4031.2
16GB RAM
454GB disk space available for paging (at the moment)
RTX 3070 Laptop GPU
AMD Ryzen 7 5800H
500x300km world size at 10m/pixel

[Theoo]

Why do you want such a large heightmap?

[HYLK]

It could very well be the case your RAM is the limiting factor.

We do not know what the inner workings are for the Flow Restructure device, but, if it needs, at any given moment, more than 16 GB of RAM for a calculation, and for some reason the program cannot use paging, it will crash.

However, I tried it on my system with 64 GB of RAM, and this is the result.

I’m sorry to inform you, but I am starting to think your request is not feasible.

[JackIt999]

Damn. I tried building it at half resolution (25,000x15,000) to see if that would work, but I got an even stranger issue this time.

My computer was successfully able to page enough memory, but the build stopped at around 70% and completed with no errors in the dialog. However, upon trying to save the output, I was met with:

“There was an error saving the File. No data was available. This is likely an internal World Machine error!”

When I monitored the build, the flow restructure device slowly made progress, but after 10 minutes or so, it suddenly jumped to 80% completion and the build stopped.

I don’t even know what I could be doing wrong at this point.

[HYLK]

I don’t think this is on you, but on WM or a library it depends on in the code. It is probably not optimised for handling such big numbers/values and crashes in the process.

Would it be an option to run the Flow Restructure (and I suppose subsequent Create Water) device at a lower resolution? That could maybe circumvent the issue, at a loss of detail.

[blattacker]

Random thought to throw in the mix: obviously I don’t know the specifics of how the devices work on a code level, but since a device like Flow Restructure is, if I recall correctly, analyzing each point as well as its relation to each point around it, is it possible that, in an extreme edge case like this, the sheer number of pixels being processed is leading to some sort of internal identifier getting an overflow error that wasn’t accounted for?

My admittedly simple thought process

Build resolution of 50000x30000 fails. 50000x30000 is 1.5 billion total pixels. If each pixel is being analyzed itself and against each the 8 pixels surrounding it, that’s 9 total operations per pixel, for a total of 13.5 billion operations (assuming that results aren’t shared across pixels). If each calculation has an internal ID for the results, the number of IDs would exceed the maximum value of an unsigned 32-bit integer. The half resolution did not build correctly (which may be a different, though possibly related, issue altogether), but did not have the fatal memory allocation failure. 25000x15000 would result in 375 million pixels, 3.375 billion operations, which would allow each operation to have a unique ID while still falling within the range of an unsigned 32-bit integer.

As a late edit: When running tests on my own machine, I’m able to successfully build a project structured as Advanced Perlin → Flow Restructure → Height Output at 25000x15000 resolution. At 50000x30000 resolution, I do not get the fatal memory allocation failure error, but I do experience the build completing successfully according to the dialog box, but the Flow Restructure device not building correctly/completely. The status box is also a color I’ve never encountered before, a very pale, almost white, green.

For reference, here are my PC specs

CPU: Intel Core i7-10700K
RAM: 32GB (16GBx2)
GPU: Radeon RX 5700XT
Page Disk: ~50GB available (none used in either build)

It’s interesting to me that I didn’t get the fatal memory allocation error, I wonder what the difference is there. I don’t believe it’s a memory issue, since @HYLK has more RAM than I do, so I wonder if it’s possibly CPU related?

Build Results for 50000x30000