Streetlights/Trees mess up tracking

Hi everyone, I'm trying to track a billboard to put text on, but in the course of the video, a number of streetlights and thin trees cross in front of it. When I attempt to track forward, whenever one of these objects comes up against the track box, it pushes the box off the billboard.

How can I get around this? I've tried single and double point tracking.