How to...camera in motion while tracking?

vanya_k
vanya_k Website User Posts: 141 Just Starting Out*

I have a problem understanding WHEN should I use HitFilm Double Point Tracking and WHEN should I use Mocha planar tracking. I have a pro version so Mocha is available to me but still I know HitFilm's tracker gets the job done most of the time.

I have a scene where I have animated 3D character running through an actual space around my house and it's shot with my iPhone and camera is suppose to move sideways with the character. So how do I track a footage that's moving and how do I make that character be on the ground at the same time?

Thank you

Comments

  • Triem23
    Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 19,833 Ambassador

    Hoo, boy, this is a deep question.

    OK, since you have Hitfilm Pro you have Mocha AND Foundry. We'll talk point tracking first, then Mocha.

    Let's start with one point tracking. You track a point of contrast. Obviously. With a single point the only information that can be extracted is two-dimensional position (X and Y Hitfilm axes).

    With two point tracking you can extract X and Y position information, a single axis of rotation (Hitfilm Z axis) and SCALE. Note this is scale and not any kind of 3D depth. It's still a 2D track. Scale can sometime work well enough to look like you've gotten 3D data if the camera is moving straight forwards/backwards. There are a couple of ways to semi-fake 3D positioning from a 2D track. Those are specific use cases, and don't sound like what you describe. Still, I recommend looking at the "Heads Up" masterclass for Hitfilm 4 for a great trick!

    With single and double point tracking, of course it's you setting the points. But, for 3D data from a point track you need at least three points. Those three point define a single triangular "polygon." And that leads to Foundry.

    Foundry's camera tracker is point-tracking on steroids. Foundry is automatically picking points for you, but it's tracking hundreds, if not thousands of points throughout a video and using that point cloud to generate 3D data. Still, the underlying principle is the same a single or double-point tracking.

    The thing about Foundry is it has no idea what the video image represents. The computer doesn't know distance or scale. It's just looking at millions of colored dots (pixels). Foundry is just trying to identify and track points of contrast. Once it has the point cloud it uses the relative movement of all those points as they move around each other to create a 3D space. The fact that it works at all is incredible! Since Foundry is tracking points moving relative to each other, it should be noted that there are literally infinite viable solutions to a single point cloud and the tracker is just going to pick one and go with it. The explains why Foundry tracks (actually, pretty much and and all camera trackers) might give you an "upside down" world or a world where the distances are really huge (points a million pixels apart) or really tiny (the entire "world" fits in a 1000x1000x1000 box).

    So, Mocha is a planar tracker. Instead of tracking points it tracks "patterns" of pixel texture. With Mocha the artist then defines the rectangular surfaces. Deforming the corners of that rectangle let's mocha try to calculate perspective. To create a 3D space Mocha then tracks the movement of those rectangles relative to each other. Like a point tracker, Mocha doesn't know what anything is, or distance or scale. Like a point tracker there are infinite mathematical solutions to any set of rectangles moving around each other. Like a point tracker, a solve might be at an odd angle or scale.

    I wanted to go over that before we get to the "when do I" question, because knowing what the software is doing helps make that decision.

    One and two point tracks always work with a static camera, of course. Two point tracking usually works if the camera is moving left, right, up, down. Moving forwards and backwards, or arcing/circling a subject usually won't work. A rotating/panning camera might work with a two point track as long as things aren't getting closer or farther away from the camera.

    Another issue with tracking is how to deal with objects moving in and out of frame. With a single or double point track you'll have to mess around with the tracker offset so, if the tracked object moves out of frame, you try to move the tracker box to something IN frame that's about the same distance from the camera. With Mocha you'll have to set up new planes and things move in and out of frame, and also make sure those planes overlap temporally. In other words, if my camera is moving sideways and object A is about to leave the frame and I've added a track to object B entering the frame, I'm also looking for object where A and B are in frame at the same time for a second... So mocha has the data to track them relative to each other. (Side note, an often overlooked thing in Mocha is each tracked plane has In/Out points. If Object B enters on frame 125 and exits on frame 277 then I want to set in/out points at 125 and 277 so Object B isn't hanging around for the whole shot with no data - which will mess up the final track).

    Foundry is a bit easier to use if you're new to tracking. There's lots of manual things to tweak to improve the track, but the defaults work so well that often it's just clicking three buttons and waiting for Foundry to do it's thing. Mocha absolutely requires artist input. You'll have to be able to identify good, flat(ish) things to track, make certain they aren't co-planar, and have good parallax. Since Mocha is generating it's 3D world based on rectangles the artist defines, you'll need to take the time to shift the corners of the plane around correctly for accurate perspective (I've had mocha tracks come out terribly that suddenly became solid by going in and adjusting ONE plane where I hadn't gotten the perspective right).

    So if Foundry is so automatic, why use Mocha? Point trackers require POINTS of contrast. If you have a lot of motion blur going on, then you have no POINTS, you have STREAKS, and Foundry is gonna have trouble. Mocha, since it's tracking "texture patterns" does a lot better at tracking shots with blur. Both programs are great, but there are shots Mocha will track that Foundry will fail at and there are shots Foundry will track that Mocha will fail at.

    See why I said this is a deep question?

    In general, if the camera is static, or moving only up/down/left/right a one or two point track will probably suffice. If the camera is moving forwards and backwards, or moving in an arc around a central subject, you're going to need Foundry or Mocha. When to use Foundry OVER Mocha will depend on personal preference first - which one are you more comfortable using - then the actual shot... Too much blur and Foundry isn't a viable option.

    I won't cover the "How to use" for Foundry and Mocha. There are tons of tutorials on YouTube, and there's documentation for both trackers. Foundry has documentation in the Hitfilm manual, or the Foundry Camera Tracker AE manual works as well. Mocha's manual is on the Imagineer/Boris websites.

    There are two key things for getting your character to stick to the ground. The first is to make sure you track the floor... With Foundry you'll have the option to select a floor plane. With mocha you'll need to make sure you actually track the ground as one of your planes - remember the computer is only looking at colored dots, and it doesn't KNOW something is "the floor" until you tell the computer "that's the floor!" If you haven't properly defined the floor for the computer, then getting your character to stick is almost impossible.

    The second key thing is moving the anchor point of the character to the bottom of the feet. Reminder: The "anchor point" is the "imaginary center" of an object/layer. Moving an object to a postion moves the ANCHOR POINT to that postion and brings the object with it. Rotation and scale is centered around the anchor point. if you move the anchor point to the feet then, once you align the feet to the ground if you need to adjust scale of the character, or spin it on the y-axis, it will stay stuck to the ground. If you leave the anchor point in the default center of the model, then scale adjustments will throw off the alignment.

    I know @FilmSensei has at least one tutorial where he shows lining up the anchor point with a character's feet. He'll find it faster than I will, and, since Jay is all kinds of awesome, I'm sure it'll show up in the next day. ;-)

    Hopefully this will help clear up the question a bit? Tracking is more an art than a science. Without seeing a particular shot it's hard to judge if a two-point would cover it, or if I'd use Mocha or Foundry.

  • FilmSensei
    FilmSensei Moderator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,067 Ambassador

    @Triem23 @vanya_k Yes, you definitely want to line up the anchor point at the bottom of the model (i.e. character's feet) to get it to track correctly on the ground plane. Here is a video that I made where I do this...

    Go to 6:34 minutes to see how I execute this technique exactly.

  • vanya_k
    vanya_k Website User Posts: 141 Just Starting Out*
    edited May 2020

    @Triem23 Wow that was the best lesson on tracking I've could have possibly ever received. Thank you so very much on this super thorough explanation. Actually I came here AFTER watching HitFilm Sensei's tutorial and attempting something simiar. I am using a Mixamo character of Ninja and my goal was for him to move and also camera to follow him.  I wish I knew how to upload pics all of this would be easier to explain...anyway my animation of him was moving (in motion left to right) so I realized ok he needs to be moving as an FBX file too. Meaning his feet are implying motion within the animation but I still have to make an actual FBX move which will give impression of him walking through my house. Now I have the worst scenario where I slowly tail along his side as he is moving so both 3D model AND physical camera are moving. I told myself this is a good lesson for a car chase scenario or Jurassic Park kind of a scene where camera is moving while CGI things are also moving. So that's why I pick this convoluted scenario rather than simplifying the scene. Anchor Point, yes  I will do as you said by either parenting it to a point or using its Anchor Point and planting it at the point of is contact with the surface.

    So far I've used Mocha only for Rotoscoping and was gonna start learning planar tracking but yeah I totally forgot about Foundry.  I think Foundry appears to be more complex than Mocha to me with all the lighting dots and Scene Resolving. I gotta look into that more. But yeah I definitely did watch a LOT of tracking tutorials but I need to read the manual too. Possibly I need to start making my own tracking points. I mean my Ninja is climbing here, hanging of my desk (he is a GI JOE figure size proportion to the real world) and I realized, there's NOTHING to track here as it's just desk. And I didn't want artificial objects in it just for tracking. I assume you PROs just add markers and than delete them in post but I was looking for a band aid, quick fix but that's not how VFX work so I need to make my own markers. Also would black tape in X or PLUS sign work as a sufficient tracker? I assume black is good for popping out and contrast and X or + symbols are good as you can notice switch in perspective and SHEAR respectively as well as scale due to their very characteristic shape. I assume that's why they use them in the movies to track background. Also I noticed circles with checkerboard like pattern. Those I encountered too. 
    Well I screen shot your lesson @Triem23 so I will keep reading it until its burned into my memory :)

    @FilmSensei hey dude I absolutely love your tutorials. Have been learning quite a lot from them. Obviously I have yet to get YOUR results but attempts are there haha.  I will go rewatch these few tuts you have on tracking and Mixamo characters as well as Foundry one you have that I've already seen few times but it's bot until I actually TRY to do something that I realize " damn looked so easy when he did it":) same thing every time. Looks so easy when you guys do it. 

     

     

  • FilmSensei
    FilmSensei Moderator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,067 Ambassador

    @vanya_k Take a look at this article from the folks over at Action VFX about tracking and tracking markers. It is an excellent article...

    https://www.actionvfx.com/blog/should-i-still-shoot-with-tracking-markers

  • pinthenet
    pinthenet Website User Posts: 150 Just Starting Out*

    Thanks for that detailed explanation @Triem23 - it's great to have all methods summarised in one post. It also explains why I have so much trouble trying to get the Foundry Tracker to track my videos, which were specially filmed using the 180° method, and resulted in lots of motion blur. It seems to be an argument for filming at a faster shutter speed and adding motion blur during post-processing?