Give him his father’s day break then leave him with the baby for his days off. No excuses, no denial accepted.
If he protests, remind him that he treated you like shit on mother’s day and still demanded his 4 hour break and that taught you that you have to demand a break to get one so you are. Buy bye. Your turn to parent for a couple days. And in the age of google and you tube, you’re also not accepting any weaponized incompetence bullshit.