Working inside of Macros isn't working properly

[Jan]

Hi,
In our production (big open-world franchise) we haven’t been able to use any WM version past 3029. Mainly this was because the broken tile-importer but also because working inside macros stopped working properly in 4xxx.

Now that the tile stuff seems to be working I’ve spent some time investigating 4045 to see if it is suitable for our production. But unfortunately it is not.

In order to structure our scenes well enough (they get complex) we structure it with macros and output the result down the network. For instance “core shapes” > “underwater” > “cliffs” > “surface detail”. This makes the top level very easy to navigate and understand, and if I need to tweak some settings it’s easy to step inside the macro to do the work.

But… In the 4xxx versions, this isn’t working:

First of all, just selecting a node inside a macro, and then generate to that device (yellow button) forces the entire macro to re-generate. This alone invalidates our scenes, since a macro can be quite intense, and also have nested macros. I can’t wait 20-30 seconds when just tweaking a blur or adjusting an expand. It should only take milliseconds. It’s a blocker.

Secondly, often I only get a 256x256 preview resolution when generating to a device inside a macro. This is not always the case though, and I’m not sure why.

To reproduce: Create a macro in a simple scene. Go inside and set up a network of nodes. Make sure to have some erosion in there. End with a simple blur device or something. Select the blur device and generate to selected device. Observe render time and preview resolution. Now tweak the blur settings and repeat the generation to the device.

Expected result: It should only take milliseconds, and the preview resolution should be the same resolution as your scene/extent.
Actual result: The entire macro is regenerated. And likely the preview resolution is low.

(I’ve been trying to get around this but putting everything in the top level, but the new versions is just too slow with a lot of nodes.)

I think 4045 is a great update overall for simple scenes, but I would really like to be able to use it on our next project. I’m really sad that we can’t take advantage of the speed/quality improvements of the newer versions.

Cheers,
-Jan

[WFab]

@Jan

As someone who had the same issues working for a well known animation facility almost 15 years back (they relied heavily on macros as well)…

Use “Blueprints” instead of macros for repetitive tasks if you want to keep your complex projects editable.

Macros keep only their outputs after a build, all other data is dumped. Makes your builds slug through their internals every time.

Blueprints are like “graphlets”, you can save a portion of your graph for reuse. This should solve some of your use cases.

For the rest, @Stephen has talked about this topic before. He may elaborate on it further, so do keep following this thread.

[Tomatiy]

In last update i did read a string about that @Stephen did make macroses to rebuild only edited nodes inside of a macroses. Maybe it means that macro will rebuild only that one device inside, if you will edit it by using parameters interface of macro, and not by direct editing of that device inside of macro by hand?

[Jan]

Thanks for the reply!

I just want to point out that this is a 4xxx issue and it works fine in 3029 for instance.

It’s like the 4xxx process can’t see inside a macro anymore. When triggering a build-to-device inside a macro and looking at the render progress, one can’t see any of the devices inside the macro, only the macro itself in the context of the top level. So it seems like a macro in 4xxx can’t be partially generated.

In 3xxx when generating a macro from the top level meant that yes, the devices inside wouldn’t keep their individual outputs, which is good and an important part of keeping memory in check. But now when I try to have everything in the top level I’m juggling well over 100gb in memory and it’s just not very efficient.

Blueprints won’t help. We use macros mainly for organisational reasons, as opposed to avoid setting up often-used network snippets. (Our goal is to keep the network clean and readable, to easily swap out and compare sections of the graph, to keep memory and overall editor UI performance in check).

Cheers!

[Stephen]

Hi Jan,

Thank you for the bug report! I can confirm the macro rebuild situation you describe. Given that 4046 Final was released yesterday, I wish I had learned of this issue months ago But I’ll see what I can do.

Essentially, macros operate in either

1. Always-conserve-memory mode, or
2. Whatever the world is set to, when the macro is opened for editing.

Normally the triggering device (white) should only invalidate its following devices, but you can see that the Parameters and Input devices (and their dependency chains) are marked as well for rebuild.

That behavior you’re seeing is designed to ensure correct rebuild behavior in the face of memory conservation mode. It should turn off when the macro is open, but it is not.

[Jan]

Thanks Stephen, it would be awesome if this is fixed! And if you’re active on the development, I can submit some suggestions that would also help us out . (“Instanced Inputs” is probably the main one, that would simplify our work the most. We’re spending a lot of time keeping the network clean and legible. But funneling inputs all through the network is time consuming. It would be a lot nicer to have all our file inputs in one group to the side, and just copy the inputs as instances to where they are needed in the graph. Same thing for macros, but needed to a lesser extent).

For now I’m aborting my evaluation of 4045 and our team will need to go back to 3029. But I hope we can upgrade in time for the next project!

Keep an eye out for our game, for which we used WM a lot on the terrain. It is released on March 20!

cheers,
-Jan

[Stephen]

Just an update that the fix for this is one of the changes in the 4047 bugfix release.

Jan, would love to hear more details about game release - What is the name?

[Jan]

cheers Stephen, I’ve downloaded the 4047 version and will have a look. I’m in the middle of something else now so it might be a few days before I can look at this again. Thanks again…

About the game, we’re not supposed to talk about it before release. But…

Assassin's Creed Shadows: Story Trailer

[Tomatiy]

i did hear story that Odyssey map was made by using WM by one environment artist girl (or he, or they, sorry)
but now i’m surprised

[Jan]

Hey Stephen, I had the chance to test it today. It seems that it works but only one level down.

In nested macros (a macro inside a macro) the issue is still there unfortunately. And we do have quite a few nested macros throughout the network in order to be able to switch out components inside bigger components. For instance, a macro that spawns trees (for visualisation only) is a fairly small macro but it is copied a few times inside another macro for different types of trees. But they are all individually tweaked (which is now a terrible job). Other examples are cliff structures, underwater sand patterns, etc, etc…

Is it possible to fix that?

4047 (as long as I don’t find something else) might not be a dealbreaker anymore though. Some macros will become a lot bigger inside, if I have to bring everything into “sublevel 1” from having been in “sublevel 2”, or even “sublevel 3”. But at least the top level should be cleaner and more responsive to work with.

Any thoughts? I would like to avoid having to rebuild everything, and lose flexibility, if I don’t have to!

Cheers!

EDIT: I might have been premature… I’m not sure what is going on but when testing with simple nested macros it seems to work, so I need to investigate a bit more with the content I’m trying to convert. BUT, if you think there might be some issues lingering with nested macros, feel free to let me know!

EDIT2: Some strange stuff is going on. Even just one level down I sometimes get a complete rebuild of the entire macro. But I can’t find any reliable repro case! I’ll spend some time tomorrow as well to see if I can find some consistency in a clean scene without any legacy devices/macros from previous versions.

[Stephen]

I was going to say - the changes shouldn’t be sensitive to nesting. However, the behavior IS sensitive to if the macro is open or not. For example, if you have the top and a deep-level macro open, but not the intermediate macro - it will trigger a complete rebuild of the intermediate macro (because memory is being conserved within it)

Edit:

I should also mention that you will see a rebuild occur when you first open the macro and then change something, because it does not have internal results from prior builds. Once the macro has been rebuilt once it should do the expected thing.

[Jan]

Hey,
I’ve been working with the 4047 release a bit. And while it seems like it is possible to work inside macros there are still some weirdness going on.

In any case, I just installed the latest (4048) and I have set aside a chunk of time to work on the erosion recipe over the next couple of days. So hopefully I can track down some reprocases, if I run into some issues.

One thing I can say, off the bat, is that the first time one wants to generate to a device inside a macro, the entire macro builds. And this can lead to some issues if there are broken (in progress) bits or if the ouput pin isn’t connected for instance. Example:

In this example I’m inside the Coreshapes macro and wants to generate to the selected device inside the macro. And the process goes through all previous devices including the macro itself. And if there is something in the macro that isn’t working (downstream) there can be some issues. And this is a likely scenario since I have a lot of work-in-progress network snippets etc in the macro…

Surely, the intent is to generate to the macro, and then step inside it to “Macroname_In” rather than generating the entire macro first?

Also, it would be good if the macro devices (in, out, help, params) were also renamed when the macro is renamed. As has been the case in previous releases (3029).

Cheers,
-Jan

[Stephen]

Ah - yes, you are right. My test cases included working on unrelated devices inside the macro, but did not include having later devices that break the macro.

Behind the scenes, the problem is that Macros have always “owned” their insides. When the top-level macro representation is activated, it retrieves its inputs, performs internal network calculations, and sets its outputs. This works great… until you want to work inside of a macro fulltime Since the transfer of data from the outside to the inside is performed by the macro itself, it needs to run. This causes both of the issues you mention.

The builder understands now how to step inside the macro and do a partial build across the container edge, but the macro device itself first needs to transfer the data to the inside input/param devices. If the macro has already transferred its data to the inside successfully, everything works as expected. Both of your reported issues occur when it has not yet done so.

I will have to think more on this.

A workaround:

Why did this work before? Oddly enough, because of a bug in the macro! If an output is flagged as “Required”, macros now will report a top-level error if it is not being transmitted from the inside. They used to ignore the flag and report success regardless.

This suggests a workaround - simply set any output ports to “optional” and the macro will build successfully even if its outputs are broken. It will still build the entire inside on the first go, but there will no longer be an issue with in progress snippets failing the build later.

[Jan]

Hi,

I found a reproducable case, but I’m not sure what is going on, or exactly which device is causing this. I noticed this when I wanted to use the Rock and Soil macro do colour some base layers for visualisation. And I just couldn’t get it to generate at all inside of a macro…

Repro:

1. Open the default scene
2. Select the Rock and Soil Macro and build to device
3. Observe how it will generate properly and displays at full resolution in your 3D window
4. Select the group containing the Rock and Soil macro, right-click and convert to Macro
5. Enter the macro, select the Rock and Soil macro. Build to device…
6. Expected result here should be the same as before. But instead the macro isn’t being built at all. (nothing in the macro is).

Undo back to start, or create a new scene…

Now, if I leave the scene view, occlusion and layers devices out of the group before converting to a macro, it works as expected. But as soon as I include the “layers” device inside the macro, it stops working. And while it is a little inconsistent in other scenes, it is like this everytime in the default scene.

At the very least, there’s a “can’t build” error in here that needs some clarification, even if I’m doing something completely wrong :).

(Disclaimer: When converting a group like this to a macro, it is kinda hard to keep track of the resulting inputs/outputs. I usually do not convert groups to macros like this. If I create a macro I generally set up the known input/ouputs first with checkpoints and convert the core network that I’ve already started on.)

[Stephen]

Hi Jan,

Thank you for the detailed repro, that is very helpful. I will investigate deeper, but in the first confirmation test I see the same result, where the new macro output device is red and reports this error:

If I fiddle the allowed packet types for the texture output, it will start working again if I change it to “Any” or “Material”.

When the Convert to Macro command runs, it takes the port datatypes from the other side of the link and sets them on the macro output. I believe what’s happening is when that command is run on a device that has multiple output types, the automatic port type assignment is failing.

This is definitely a bug, but it appears that it is simply an import bug, not the actual operation of the macro. I will look further however to make sure.

[Jan]

Yeah I got similar erroneous connection error. I had a combiner that all of a sudden refused to accept its HF and constant connections.

There seems to be some memory/cache/pointer issues. When working in macros, I routinely need to switch between extents to clean the memory pipeline, since devices often just stop building. Sometimes I even need to restart the entire app.

Here is another reproducable issue related to macros:

The group feature breaks two levels down:

1. Create a macro in the root level, go into it
2. Create another macro inside the first macro, go into it
3. Select a node and ctrl-g to group it
4. Double-click the group

The expected behaviour is that the group properties should open and that the user is able to set name and other properties. But while the properties window opens, you can’t do anything with it.

I noticed this trying to organise my scene, which is a lot more complex than the default scene. In my scene (not reproduced in the default project), this also made the program behave badly. For instance, I couldn’t zoom the network or select other devices once I had double-clicked on a group two levels down. Only restart seemed to fix that.

[Stephen]

That’s a weird one! Thank you again for the easy repro steps - it makes it very easy to fix things like this. I’ve fixed this group issue for the next maintenance release.

When there’s a logic bug such as the above, sometimes WM will begin behaving oddly. I believe this is the “devices stop building” bug you mentioned.

The root cause of this is WM believes a preview world is in progress when none is (perhaps because there was an uncaught error that cancelled it). One of the prime symptoms is if you try to move a device in the workview, but when you let go of the mouse, the device snaps back to where it was.

These are obviously pretty important issues to track down! But in HR you should be able to “re-synchronize” that state by performing a build, clicking the Preview button in the statusbar, or performing undo/redo.