It's pretty well known now that a lot of Western and Asian countries have very low birth rates, in many cases too slow to maintain the existing population.

We often hear this talked about as a "population crisis" and countries like Japan are taking measures to massively increase immigration to counter the lack of local births.

So my question is, why does it matter? So a country has 20 million people this year and may have 15 million in 20 years. What's the problem with that? Why does it need more people?

If one of the major reasons for low birth rates is the inability to afford kids, then wouldn't losing a quarter of the population make housing prices plummet to the point where basically anyone could afford one? Then they'd be able to more easily afford kids and the population could stabilize.

It seems to me that if people aren't having enough kids, it's a sign not that the country needs to find other ways to grow the population, but that the country can't support a larger population and NEEDS to shrink.

Edit:
Lots of interesting responses here. I didn't expect so much interest in the topic. I've got some interesting links about economics to look into from some comments. Particularly regarding South Korea and their population collapse.