my bf and i average 4 hours per session which is twice a week !! you're definitely not spending too long at the gym--we are! haha. but I dunno; between a top rope, bouldering, and weight lifting, I often feel like 4 hours twice a week isn't enough! and we always climb until we start noticing a decreased ability.
i think the only reason you should be concerned is if you start getting hurt.