It isn’t because of Gwen’s screamed criminations, or the accusing stare from Owen. It isn’t Tosh’s quiet acceptance that he doesn’t deserve. None of those things make him regret it.
No, it’s the look in Ianto’s eyes. It’s that look that makes Jack feel guilty in the end, because he knows it too well. He’d worn it in himself after the Doctor left him. Of all people, he should have known better than to abandon anyone after a life
Ianto would eventually chase away most of the blame Jack placed on himself with cool logic and warm lips, but that look in his eyes… Jack would never forgive himself for putting it there.