Any way to make masks look smooth and not jagged like this?

klauxgmd
klauxgmd Website User Posts: 13 Just Starting Out*

I use masks a lot in my videos, but they always look jagged like this, even in final renders, is there any anti aliasing tool for masks, or is this a glitch, or are they really meant to look like this?

(notice how jagged those cyan curves look, they're from a mask)

Best Answer

  • Triem23
    Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 19,969 Ambassador
    edited September 20 Accepted Answer

    Yup, looks like a Bug or Glitch with the Expansion and Roundness mask parameters. @TheBenNorris @Ady Ben, Ady, you two can skip right down to the big headline, below. Or, read the entire post and watch me speculate wildly! That's gotta be SO much fun for you!

    @klauxgmd Question. For a round shape, why are you using such a roundabout method to create a round shape. Why not just start with a circle mask?

    Here's what I THINK is going, on, but, bear in mind I'm not FXhome Staff, or a Hitfilm dev. I've tagged in a dev and QA lead, and will add an "informal bug report."

    So. When creating a plane layer, obviously a pixel grid is being created. The mask is defining a vector curve that's clipping to the pixels. It LOOKS (as an educated guess) like - in Hitfilm's order of operations - the mask is being rasterized before being passed onto the controls in the SHAPE tab. And, while the Expansion and Roundness values go up to ridiculous high values, no one was REALLY anticipated to use those values. After all, if rounding by 2000 pixels in a 1920x1080 comp you're rounding the mask by a larger value than the timeline. Same goes for expansion. Why expand a mask by 900+ pixels all around in a 1920x1080 tineline. It's bigger than the frame.

    I believe the devs anticipated most users creating the masks at, or near intended size with those controls intended a bit more for refinement, not the unusually large values you're using.

    Additionally, the "EXPANSION" option seems to be expanding the mask edges evenly around the center, however, the "ROUNDNESS" property is eroding/shrinking the mask evenly towards the center. You can check this for yourself using a larger-sized mask. The Expansion and Roundness properties seem to be fighting one another.

    I could be wrong on the mask rasterizing immediately, since this type of distortion isn't occuring with extreme scale values - like taking mask scale up to 5000% - or using Roundness, below.

    So, the problem seems to be confined to the Expansion property.

    Ben, Ady - steps to reproduce issue in HF 16.1.11208.7204.

    • Create a Comp. I tried 1080p
    • Create a plane.
    • Add a RECTANGULAR MASK mask in a bottom corner. I had mine take up about the bottom/right third of the screen. (This happens with an ellipse mask as well, but it's more obvious with a rectangle). I went ahead an let my mask extend beyond the bounderies of the timeline frame.
    • Twirl down Shape Properties.
    • Increase EXPANSION. By the time Expansion hits about 250-300 pixels, edge jaggies are becoming obvious.
    • Increase ROUNDNESS. This actually starts to smooth out the jaggies just a bit, but it's not really helping.
    • Return EXPANSION to 0. As Expansion reduces the jaggies go away. Expansion of 0 and higher ROUNDNESS values don't exhibit the jaggies. I assume this is because roundness is eroding the edge.
    • To compare, reduce expansion and roundness to 0, and twirl open TRANSFORM
    • Go ahead and unlink the SCALE Property. Take either axis and increase to a very high value, say, over 5000%.
    • Mask edge remains normal. Increasing EXPANSION will still cause jaggies, but not any worse than with the mask scaled at 100%.

    Look like the issue is confined to the EXPANSION parameter only.

Answers

  • Triem23
    Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 19,969 Ambassador

    This looks like you've masked a plane and scaled it up?

  • klauxgmd
    klauxgmd Website User Posts: 13 Just Starting Out*
    edited September 19

    nah, what I did was I made a small square mask on the bottom right corner, increased roundness to max value, and keyframed expansion to make the circle grow

    (I also added a small feather to try and make it look less jagged, but it didn't do much)

  • klauxgmd
    klauxgmd Website User Posts: 13 Just Starting Out*
    edited September 19

    also I didn't use any planes there, the mask is in this layer outlined in red here:

    after I animate the mask for that one layer, I duplicate it behind itself and add fill color effects to make this kind of trail that follows it, that's it:

    (yellow layer is that one outlined in red I mentioned above, the other blue ones are duplicates for the trail effect thingy).

    Btw this is how my controls tab looks like for all of those layers, there aren't any scale keyframes, or any other transform keyframes in the layers themselves (just in the mask expansion, as I said previously). If the layer looks sharp, which is does, the masks should too:

    Also this is where the mask is, just to add more clarity:


  • klauxgmd
    klauxgmd Website User Posts: 13 Just Starting Out*
    edited September 19

    it still looks bad if I scale the layer back to 100% btw, and this happens literally every time I do this type of expansion mask, regardless of layer. Even when I do it in text layers, it still looks jagged, like here:

    (frame from an old video)

  • klauxgmd
    klauxgmd Website User Posts: 13 Just Starting Out*
    edited September 20

    I didn't know how to use the circle mask tool because the ellipse mask button is kinda hidden (you need to hold the rectangle mask for it to show the other options, or click the rectangle mask and then click E for the ellipse option). I'd like to suggest a slight ui change, where the ellipse mask is right under the rectangle mask, visible at all times, so it's easier to find. Also thanks, using a circle mask makes it completely smooth!

  • Triem23
    Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 19,969 Ambassador

    @klauxgmd

    Hah! The Rectangle and Ellipse/Circle mask buttons JUST GOT combined like they are in Hitfilm 16. In prior versions they were different buttons!

  • klauxgmd
    klauxgmd Website User Posts: 13 Just Starting Out*

    Oh, so that must be why I recall using it in previous versions, but never found it in Hitfilm 16 lel