I wouldn't have let them punish you. Had someone found us I would have bolted slowly enough to have been caught and say I came to beat you up. Having earlier punched Tony, they would assume I was in a state of mental distress and dealt with me without further involvement of you. If asked how I got in, I'd say I observed someone entering and learned the key. Nothing would lead back to you or anyone else. No worries.