Not sure about the ” perfect life ” bit either, wasn’t he meant to be shagging Mary Magdalene on the quiet?
It that was true rest assured they’d be an excuse for it.
The excuse is probably related to his lack of a father figure. I’m sure Joe tried his best, but jeez’s true dad was either Gabes (who seemed to disappear before the birth) or he was his own dad (who only wanted him, so he could forgive other people’s sins, and make very little sense)