Jittering Track Points

torburwell
torburwell Posts: 2 Just Starting Out

I just got CamTrackAR and have been disappointed in the jitter in my scene once I bring it to blender. I have checked and it is not the footage that is shaky, but only the tracking points that I put in the scene while using the app. If someone could help it would be greatly appreciated.

Comments

  • JimmyGunawan
    JimmyGunawan Posts: 2 Just Starting Out

    I've experienced the same issue. I have a feeling it's a bug with Blender import. I don't know if it's the same wobble / jitter when the tracking data is loaded into After Effects.

  • keithham
    keithham Posts: 3 Just Starting Out

    I've had the same jittery issues in After Effects. It's hard to say if it's an FXHome issue or an ARKit issue. I've played around with Apple's AR before and the models you place do drift or jitter some as you move your device around. It's not like if you have a good track from After Effect's native camera tracking in post. I think it's a limitation of ARKit and the hardware on your device. I would think a device with LiDAR (which I don't have) would work the best.

  • GrayMotion
    GrayMotion Posts: 1,632 Enthusiast

    I have an iPhone 12 Pro Max...jitter is pretty prevalent across the entire video actually. The model is locked on the point but the entire scene is shaking. . It looks like the hand held action is just too sensitive? I also noticed in FXHome's demo video that the entire scene also jitters.

    What you think? Maybe locking down the iPhone on a tripod and then film pans and tilts will eliminate that jitter?

  • Triem23
    Triem23 Posts: 20,275 Power User

    Phone sensors are tiny little rolling sensor CMOS sensors and are subject to serious wobble issues (my guess is, if you look carefully your footage is actually distorting and microstretching), additionally, while I'm not an iPhone owner I believe you're getting VFR footage. Both rolling shutter wobble and VFR will cause drift. Additionally, the core tech - the Apple ARKit - has its own limitations. It's ARKit that limits the footage to 1440p not 4k.

    None of this is the fault of FXHOME, and none of this is anything FXHOME can fix. CMOS wobble is inherent in the hardware and can only be fixed by using a global shutter chip (to my knowledge the iPhoneXs was the only iPhone with global shutter (update - global shutter for the IR sensor, not the main camera). Timing issues caused by VFR can only be fixed by actually recording CFR. Again, this is an issue for Apple.

    Using a gimbal will eliminate a lot of jitter. Using a tripod will eliminate more. Even then the very act of panning, tilting or moving the camera causes rolling shutter distortion. Small sensors (phones) have much more distortion than large sensors (Cinema camera).

    The tracker assumes everything is still. Rolling shutter distortion reads to a tracker as MOVEMENT and it messes up the track. VFR changes the cadence of movement and throws off the track.

    In other words CamtrackAR is probably performing as well as the limitations in the underlying tech allow, and there ain't nothing else FXHOME can do.

    For those who don't know what rolling shutter is, I leave this article.


  • GrayMotion
    GrayMotion Posts: 1,632 Enthusiast
    edited November 2020

    Thanks for that Mike.

    I just wanna add that I have seen pretty solid tracking using the iPhone AND a camera (using an Elgato HD60) inside Unreal...defiantly no " jitter" in the footage. Solid tracking.

    I'm going to do some testing with Camtrack AR though. Everything I've done so far is fairly close to the camera. Nothing too far away (+10 feet). I'll get me a hand-held device as well as another smaller Manfrotto tripod w/ball head (Compact Action Smart) and do some shooting and see what I can see... (um, See what I can see - sounds like a mechanic working on a car.)

    Anyhoo...I noticed the first AR video that FXHome showcased in Jul 31, 2020 they are using a hand-held device and there is very little "jitter". The second video released Aug 20, 2020 - the intro - you can defiantly see the jitter in this entire shot. Now the video released Oct 29, 2020 the jitter is hardly (if at all ) noticeable. So - rig difference to be sure - I'd imagine :-)

    Thanks again bud 👍

  • fxhomer48774
    fxhomer48774 Posts: 1 Just Starting Out

    I´m testing the CamTrackAR app here also and i notice that i can´t get stable tracking!! when i import to blender i always notice Jittering in the Track Points!! and with this the final result is not good because the 3d objects will look flying! I did a comparison, doing a quick tracking with the blender motion tracking with the exact same video and doing there get always a perfect result. Is a bit disappointing because the apps seem to be nice but can't offer a good result to using it as showned

  • acestu
    acestu Posts: 8 Just Starting Out*

    Hi,

    I have just posted with the same thing no matter how i try once imported into Blender even using the latest importer I get the jitters and anything placed in the video just does not look any good

    Thanks

  • spydurhank
    spydurhank Posts: 3,156 Expert

    Howdy @torburwell, I'm Frank.

    Been doing professional camera and 3D object matchmoving since 2008 plus I code my own version of Blender to work with Hitfilm. 😀 Let me just tell you that I learned 3D camera and object tracking by hand with no tracking software, I had to transform and keyframe a 3D camera to live action shots and come up with my own calculations to get correct camera data like, lens, sensor, depth of field, focus distance, and motion blur. Had to do all of that stuff in my head and by hand... Dang. 😀

    I've seen a bunch of test phone videos tracked with Camtrackar on Youtube,

    Lots of em' have Gimbal lock and also jittering but it really seems that most people either don't even see it happen or they simply ignore it hoping that no one notices.

    Let me give you the straightest answer because in this case... the issue can only ever be one of four things when you get a bad camera solve... one is that there are not enough tracking points for Camtrackar to get an accurate camera solve, too much motion blur and rolling shutter, and the other issue is that the Camtrackar script is not importing the camera data properly to begin with.

    The only way to fix Camera jitter and Gimbal lock is by providing the software with more tracking points. There is no other answer to fix Camera jitter and Gimbal lock because... the answer should always be... you need more track points... unless the footage was simply just shot in a bad way in which case... re-shoot the scene but be a bit more prepared so you can cut down on Motion blur and Rolling shutter.

    If Camtrackar has the option... you should give it as many track points as you can while matchmoving a scene. But if Camtrackar only exports a limited number of points, like less than 8 points at a time, well... then you're gonna have to plan your shots accordingly to avoid as much handheld jitter as you possibly can and avoid rotating your phone over 90 degrees in any axis. Still, remember to use as many tracking points/markers as you can... the more the better.

    In Blender which you obviously know, has a dedicated camera and object tracker, you can get away with as little as 8 tracking markers as long as they are visible throughout your video clip. If you get a bad camera solve then you can easily just add "More" tracking markers. The more points you have the more stable your camera solve will be. If you have too much motion blur and rolling shutter... then the built in Blender matchmover would give you a bad track just like Camtrackar. Try it in Blender and see for yourself.

    Same thing with the Hitfilm Foundry camera tracker... if you get a bad camera solve... simply raise the amount of tracking points/markers that the Foundry Tracker should use. If you have too much motion blur and rolling shutter... then the Foundry tracker should also have problems like Camtrackar. Try it with the Hitfilm Foundry tracker and see for yourself.

    This is how I deal with camera sensors, rolling shutter, and motion blur from mirrorless cameras "more tracking markers" and it works unless you've got some really crazy and chaotic movement going on which causes way too much rolling shutter and motion blur. Too much rolling shutter and or motion blur will for sure produce a bad camera solve if say for example... Camtrackar only exports a handful of tracking markers in which case, it is simply not enough to get a proper camera solve from 1, 2, 3, or 4 tracking markers where the scene just has too much movement, then it'll absolutely go crazy if you rotate your camera over 90 degrees in any axis which will rear its head up as Gimbal Lock.

    If adding more tracking points/markers, still doesn't work for you while using Camtrackar... even after you re-shoot your scene and really cut back on rolling shutter and motion blur... then the issue is indeed with the Hitfilm Camtrackar import script for Blender. I'm saying this to be polite because you should always try adding more tracking points first just to make sure because then you will be absolutely, positively sure that it is not your footage or your phone sensor which you only need to ever take into account if you can see that you have too much motion blur and or Rolling shutter.

    And if you really want to trouble shoot this issue? It is really easy to track down the real problem.

    Matchmove your footage in different dedicated 3D applications "your choice, I use Blender" and also with the Hitfilm Foundry camera tracker or the After Effects, Foundry camera tracker. If you can get good camera solves from all of the other dedicated 3D apps as well as the Foundry tracker in AE and Hitfilm... that can mean only one thing... there is something wrong with the Camtrackar script. Duuuh... 😏 It is the only way to be sure... that way you don't have to waste your time listening to people who are just guessing in the first place.

    You gonna look to the Apple technology and other dedicated 3D tracking and matchmoving technology that has been around for a while or the Camtrackar script that is in its infancy. Yeah... I know how to track down real problems. 😀

    Later,

    Frank.

  • goji1986
    goji1986 Posts: 3 Just Starting Out*

    I found this out on somewhere out and figured it would be very helpful to post here - in Blender, motion keyframes start at frame 0, whereas video begins at frame 1. Select all the camera keyframes and shift them over by one frame and you'll be all set. Tried this myself after some frustrating test results and now everything is solid!

  • makaveli911
    makaveli911 Posts: 1 Just Starting Out*

    beautfull brother, thank youy so much it work with me and iv been in this problrm so long withou knowing hot to solve it. thanks