Masking second layer for pop up clone

I have a shot where I am sitting doing stuff and another me pops up and interacts with the first me. So in composite I have the first layer that doesn't have a mask. then the second layer with the pop up me. In this second layer I have masked around me. I can only see the two interact when the opacity on the first layer is lower than 100%. There are no other effects in the scene. What am I missing?