It's a totally personal decision, but I would go ahead and do it before the interest rate and prices goes up further. Unless you are into Cali, what is out there like in midwest? Yes, DMV has hurriance and stuff like that, but nothing like other natural disasters like other parts of the country. 

What I understand is that home prices have gone up by 3 to 8 times in the last twenty years in DMV areas. One thing for sure, your area is not going to die down anytime soon. That's a fact. Not every part of MD or DC, but VA is a great place to raise kids. 

So, choice is yours....big house in a big lot, or active living (for you and your kids) in a smaller house.