Whilst reading this I thought to myself, fun problem to tackle. Here’s my take on it!
It all has to do with using the Abmient Occlusion device to “reveal” the concave parts of your terrain. See the image below.
You can clearly discriminate the concave parts using this method. I suggest doing some pre- and post-processing on the masking of course, and I’ll link the .tmd file with my processing setup down below.
Then, using this masking, we can create the beaches accordingly (again, with a bit of preprocessing of the terrain to help the Coast Erosion device).
On the left you have the naive approach, just using the Coast Erosion device. On the right, we do a bit of preprocessing of the terrain and drive the beach creation using the “concave mask”.
The nodes:
As you can see, I’ve downscaled the Ambient Occlusion device, this is because we don’t need a high resolution AO map, and this really improves rendering time!
Good luck!
beaches-based-on-convex-concave-terrain.tmd (130.7 KB)