I’m curious because I live in a very young, international city and basically never had this problem, but hear a lot of others in Germany mention it.
For Germans I meet, whether young, old, professional or casual, the conversation almost always starts and continues in German. If it was something important that I needed but couldn’t understand, I was always the one to request to switch. And it was the case back when my German was pretty bad, so it wasn’t related to my German level.
Just keep speaking german. You don't have to switch