Multiple masks on one layer acting weird

Hello folks, I'm doing something that seems simple but it's not working right.   My shot has a book with blank pages and hands over it. I'm trying to add some text to the page which I created as PNG in Photoshop. Of course I don't want the text going over the hands so I masked one hand out and it works great, then I started masking the other hand and it's acting crazy. Like one mask will work or the other won't or one is inverted so I click the Invert button and then they both stop working. Is there some trick to having 2 masks on one layer?