Often times, when I am building large worlds, I have plenty of different masks. Masks and layout generators and color generators usually have low entropy, mostly 1 and 0 with relatively few values in between.
Often, I run out of RAM first. I was wondering if it were possible to compress heightfields and bitmaps losslessly, on the fly, quickly.
I looking into gpu solutions, because it is often idle while building. But compression is inherently not a SIMD algorithm, and current gpu implementations barely reach the speed of a normal compression algorith on cpu.
Most cpu compression algorithms dont do much more than a few tens of mb per second per core, which is still woefully inadequate.
LZ4 on the other hand can do 300Mbyte/s compress and 1GB/s decompress. Per core
This means that WM could possibly, in 1 second decompress all the inputs it needs for a device and compress those which are not used in the next built node.
I just wanted to follow up and say thanks for bringing this up, as I’m investigating using lz4 to compress the scratch files if possible – on a non-SSD system and very large datasets the WM buildtime becomes bound by file I/O, so compression of temp files becomes a very useful property.
In general heightfields don’t usually compress that well due to lots of randomness; but masks certainly do!
I did some preliminary implementation of LZ4 for scratch/session saving.
Generally speaking the storage savings are very minimal unfortunately, as procedural terrains often contain alot of randomness that cannot be compressed well.
However, LZ4 is so bloody fast that there is effectively no overhead from using it. Furthermore, certain common types of heightfields (masks, layout shapes, etc) ARE fairly compressible, so on the whole it seems like a pretty worthwhile thing to implement.
I will probably merge this experiment into the feature branch shortly, bringing this ability to the public release.
