Wm masks tend to be smooth around the edges, which creates problems when exporting to engines that have optimizations in engine that handle alpha differently, especially when you export masks in an 8 bit format. Try to keep your masks at 1 and 0, no halfway greys. You can get away with small amounts, thin bands, for smooth transition. Use bias/gain device for cleaning up the mask. I don't have extensive knowledge of how UE handles alpha, but this is a common problem all around.