You haven't returned the rental car if you don't return the keys and you haven't checked out until you've returned the keys and relinquished access.
The other complaints against this guess are typical B quality human stuff and not worth mentioning in a review or even to your fellow hosts.
Because you people pleased yourself into the corner and handled this poorly, i recommend you mentally thank these guests for the learning opportunity and move on.
A better practice would have been, "Oh, if you'd like to retain access to your shared space I'll have to charge you for another day, otherwise I'll need the keys at checkout for the next guest."
Best practice would involve a smart lock that deactivates at check out. Each guest gets their own code. "I'm sorry, to maintain privacy and safety of our guests in a shared space listing, your smart lock code activates during your stay and deactivates at 11 am at checkout. We're not able to store luggage or allow entry after checkout."