Very slow nodes, any workaround / faster alternate way?

Bug Reports
#1

I finally bit the bullet and got myself a 256gig of ram server for my high resolution needs.
However some basic nodes aren’t multithreaded or just very slow, it doesn’t seem to be the same nodes as those that are usually slow (like erosion) so i assume it’s just nodes that scale very badly with size.

On my graph containing pretty much all types of nodes (including multiple noise / erosion / color generation) the slowest node by far is . . . blur! And what’s surprising is it’s slow even if i’m passing it a mask that is only a very very small area on my map (maybe 1/50 of the total area size).

My goal with blur is to naturally flatten areas to make for easier placing (so areas like cities) i have 2 blurs chained together (couldn’t get a strong enough blur with a single node).

Is there any hope blur would better use the mask info / get multithreaded / performance improved? If not is there a set of node i could use to get a “cheap man’s strong blur” where i end up with rather flat but still natural areas based on a mask?

#2

Hi there,

Blur is multi-threaded (at least as of 2.3), so I’m somewhat surprised that it is your performance bottleneck, even with a large radius.

I agree it would be useful for all devices, in general, to have a specific optimization for masks whereby they can skip any calculations in completely zero areas. Currently the device calculates like normal and then has its results combined according tot he mask, which is the easiest to implement but does not grant any speed gains. I will take a look at the dev requirements to do this and figure out if it will go on the add list! Also on the dev side of things, uniform blur is currently as slow as gaussian blur but could be sped up considerably – if that optimization is done then uniform would be the best way to blur large areas.

Unfortunately, to blur you will need… blur. Any alternative method is likely to be less efficient in WM.

#3

Well i just finished what i thought would be a fast (but due to blur was slow) render at 16K of a complex node graph and as you can see, blur is “by far” the bottleneck.
The whole render took around 85 000seconds, with 65 000 of them spent on blur! That’s 18 hours worth of bluring!
And as i mentioned it’s only blurring a very very tiny part of the world thanks to masking (actually, on the specific extent i was running, nothing was part of that mask, shouldn’t that node have been auto skiped?)
If you want to contact me at ronan.thibaudau@rt-informationtechnology.com i’ll be happy to go back & forth with you or give you access to my large server for testing as i think there’s definately something to be done here, every other node was fast including erosion which was reasonably fast and most of all scaled normally (i can handle long times, but i want highter res and blur just scales horribly).

The other culprit i’ve found is layout generator when using lines, it’s very very slow too and will be my 2nd bottleneck with scaling, but definately not on the same level as blur, i was also surprised at how fast some nodes were, nearly instant for some at 16K, so to enable really high res scenarios i need (128K + output resolution) i would really need a faster blur (at worse i can roll my own as a post process on the output) and faster layout generator, any chance for those 2 getting looked at in light of those results?

#4

Adding this for reference, this is the same render at 4K (vs 16K in previous post) so that you can see how things scale, as you can see erosion scales as expected unlike the blur & layout (what is expected is an optimal 4X per doubling in size, so from 4 to 16 i expected 4*4 = 16X speedup when going from 16K to 4K)
Actual speedup:

  • Erosion 19846 => 337 => 58X
  • Blur 32560 => 465 => 70X => I never expected blur to be slower than erosion, it’s also hard as i can do with a single erosion node but i’ll probably need 20 or so blur nodes
  • Advanced perlin 35 => 2.4 => 14X => Scales ok, is very fast anyway as it’s fairly simple and finely multithreaded
  • Mountains (layout generator with multiple lines / poly) 2123 => 141 => 15X => Not too bad but unexpected as it seems pretty simple, also this is the kind of tool i use to guide everything but definately don’t need very detailed (16K render looks almost like 2K for example) would be really nice to be able on those nodes to “downscale”, like to be able to say “max 2048K” or “1/4X current resolution” and have world machine either rescale it or sample it for further nodes, it would barely change the result and i’d be very happy with this being at 2-4K even if i’m generating at 128K
#5