Any way to make masks look smooth and not jagged like this?

klauxgmd Posts: 19 Just Starting Out*

I use masks a lot in my videos, but they always look jagged like this, even in final renders, is there any anti aliasing tool for masks, or is this a glitch, or are they really meant to look like this?

(notice how jagged those cyan curves look, they're from a mask)

Best Answer

  • Triem23
    Triem23 Posts: 20,280 Power User
    edited September 2021 Answer ✓

    Yup, looks like a Bug or Glitch with the Expansion and Roundness mask parameters. @TheBenNorris @Ady Ben, Ady, you two can skip right down to the big headline, below. Or, read the entire post and watch me speculate wildly! That's gotta be SO much fun for you!

    @klauxgmd Question. For a round shape, why are you using such a roundabout method to create a round shape. Why not just start with a circle mask?

    Here's what I THINK is going, on, but, bear in mind I'm not FXhome Staff, or a Hitfilm dev. I've tagged in a dev and QA lead, and will add an "informal bug report."

    So. When creating a plane layer, obviously a pixel grid is being created. The mask is defining a vector curve that's clipping to the pixels. It LOOKS (as an educated guess) like - in Hitfilm's order of operations - the mask is being rasterized before being passed onto the controls in the SHAPE tab. And, while the Expansion and Roundness values go up to ridiculous high values, no one was REALLY anticipated to use those values. After all, if rounding by 2000 pixels in a 1920x1080 comp you're rounding the mask by a larger value than the timeline. Same goes for expansion. Why expand a mask by 900+ pixels all around in a 1920x1080 tineline. It's bigger than the frame.

    I believe the devs anticipated most users creating the masks at, or near intended size with those controls intended a bit more for refinement, not the unusually large values you're using.

    Additionally, the "EXPANSION" option seems to be expanding the mask edges evenly around the center, however, the "ROUNDNESS" property is eroding/shrinking the mask evenly towards the center. You can check this for yourself using a larger-sized mask. The Expansion and Roundness properties seem to be fighting one another.

    I could be wrong on the mask rasterizing immediately, since this type of distortion isn't occuring with extreme scale values - like taking mask scale up to 5000% - or using Roundness, below.

    So, the problem seems to be confined to the Expansion property.

    Ben, Ady - steps to reproduce issue in HF 16.1.11208.7204.

    • Create a Comp. I tried 1080p
    • Create a plane.
    • Add a RECTANGULAR MASK mask in a bottom corner. I had mine take up about the bottom/right third of the screen. (This happens with an ellipse mask as well, but it's more obvious with a rectangle). I went ahead an let my mask extend beyond the bounderies of the timeline frame.
    • Twirl down Shape Properties.
    • Increase EXPANSION. By the time Expansion hits about 250-300 pixels, edge jaggies are becoming obvious.
    • Increase ROUNDNESS. This actually starts to smooth out the jaggies just a bit, but it's not really helping.
    • Return EXPANSION to 0. As Expansion reduces the jaggies go away. Expansion of 0 and higher ROUNDNESS values don't exhibit the jaggies. I assume this is because roundness is eroding the edge.
    • To compare, reduce expansion and roundness to 0, and twirl open TRANSFORM
    • Go ahead and unlink the SCALE Property. Take either axis and increase to a very high value, say, over 5000%.
    • Mask edge remains normal. Increasing EXPANSION will still cause jaggies, but not any worse than with the mask scaled at 100%.

    Look like the issue is confined to the EXPANSION parameter only.