Memory leak in tiled builds

Bug Reports
#1

Hi, im experiencing some kind of memory leak that prevents me from completing tiled builds on my machine

(WM2.15 on vista 32b , 4GB ram)
The world extents is very small and the tiles are very small too (256*256).

The tiles are actually built, but in the blending phase, it inevitably runs into an out of memory error.

It looks like in the blending phase, world machine does not deallocate the memory its using, thus leading to an out of memory error for big tile sets (20*20 or more). Also after the out of memory error, WM continues to use all the memory and needs a restart to free the resources and work again.

I’m ONLY bulding tiles, they are not merged in a single image. The problem does not raise in the building phase, all tiles are built correctly. the problem occurs in the blending phase, which happens after the build and before the writing of outputs. The first 20 or so tiles are output correctly to disk as images before the out of memory error occurs.

I’m pretty sure this is a bug, WM does not free resources of the completed tiles in the blending phase, since I see a constant increase in memory usage during the blending phase. This makes no sense as the resources of each tile should be freed once the outputs are written to the disk and there are no other nearby tiles that requires blending.

I’ve attached a simple project that produces this error :

Edit :
I’ve noticed (too late) theres another thread about this problem :

Any help will be greatly appreciated, as this problem is basically preventing me from completing any build.

#2

Are you using memory conservation?

Also: With 32bit Vista you will not be able to use 4GB ram, only 2 or 3maybe. (but this shouldn’t matter for your problem)

#3

actually it still might be a problem if the final image sise is too big for his memory…but it shouldnt with that device chain. Most of the time i ran into this problem on my old machine was with very complex networks. You know that by using such small tiles your going to get alot of blending yuk. Its a simple enough device so why dont you try using larger tiles and if you still cant merge them…do it post in photoshop…its going to take longer to render but its worth it.

as for the 3gb for an application in windows…its called a 3gb switch…not that hard todo either.

#4

Yes I’m using memory conservation, but I tried both ways.

Also i’m ONLY bulding tiles, they are not merged in a single image. The problem does not raise in the building phase, all tiles are built correctly. the problem occurs in the blending phase, which happens after the build and before the writing of outputs. The first 20 or so tiles are output correctly to disk as images before the out of memory error occurs.

I’m pretty sure this is a bug, WM does not free resources of the completed tiles in the blending phase, since I see a constant increase in memory usage during the blending phase. This makes no sense as the resources of each tile should be freed once the outputs are written to the disk and there are no other nearby tiles that requires blending.

Please let me know if I can provide some further information for this problem.

#5

You might be able to work around this issue by setting your pagefile to a large value like 16 gigs… Its not a nice solution and I too think this is a bug - but it works.

#6

I tried this already, with a swap file of 32 Gbs, but it still runs in out of memory when it runs out of RAM, no matter how big the swap file is.Dont know if it is some limitation of vista, im gonna try on a XP machine as soon as I can.

#7

OK well because you are not merging them anyway…you could simply limit the number of tiles its outputting…then do the rest next time around. To me it makes sense that the memory increases during blending because as more tiles are processed for blending the more blending info it would need to hold for the other tiles next to them…I don’t know how its written in code…I did try your example file on my 64bit machine here and the memory consumption goes up to about 3.4GB…so its just too much for your 32bit OS…which is why there is the 64bit exe with worldmachine. But anyway i would just limit the number of tiles being output…Ive also had to do this on my old machine at times

#8

No it does not make sense to have memory consuption constantly increase during blending. It should only increase up to the maximum number of tiles used in a blending operation. If the memory consuption is normal, then it means that a blending operation involves up to 30 or 40 tiles and I doubt it.

If on your machine my test runs fine, try monitoring the ram usage and you will see that in the building phase it is constant around 500Mb (I dont remember the exact number) while in the blending phase it slowly rises up (to 3.4 Gb as you say).

Also i cannot reduce the tiles number, as it would reduce the terrain resolution. Since I cant reduce the world extent anymore, the only way to have such resolution is to increase tiles number.

#9

what i mean by reduce the number of tiles is…to output lets say 18 at a time. Normally there is a check box that says always build all tiles…but this can be switched off and you can select a range of tiles…This way you can output just a small number of the tiles or all the tiles when you do a build.

Say for example you have a world with a world extent of 4 x 256 tiles, these tiles are represented by a tile number 0,0/0,1 ect. If you select a range from 0,1 to 1,1 you will get the top 2 tiles. This will work Ive also used it before when I ran out of memory. Just a tip use the overview mode to show the tile numbers so you can easily see which ones you want…

#10

Gonna try that thanks.

#11

I managed to run a full build by splitting up the tiles as suggested by Nav. My build consisted of 4096 256X256 tiles that I splitted in groups of around 400 tiles each when building. However I had to restart WM after every build, cause I keep experiencing the memory leak problem , even with successfull builds. IE : If I run two builds of 400 tiles one after the other, the first is successfull the second one runs into out of memory error. If I restart WM inbetween, everything works.

#12

Cool that it worked…and I guess you kinda prove the point that memory doesnt come back…since you have to restart wm…but I guess that no big deal seeing as when you do a tiled build it doesnt keep the data from the network anyway. My only fear for your worls is that with so much blending across the tiles you will notice it…i know it takes longer to render bigger tiles but if you find the result isnt seemless enough try some bigger tiles…

#13