Buying a house as an investment

One thing I find interesting about living in SF is that it's the only place I've lived where the "I bought a house and it was a great investment" people who tell me that I should buy a house haven't been very wrong (historically; they could be wrong in the future, who knows).

Everywhere else I've lived I hear people say things like "I bought this house ten years ago and it doubled in value, you're throwing money away by renting", but whenever I've looked at my (very boring) portfolio, it's beaten that . A nominal (i.e., non-inflation adjusted) doubling in ten years is 7% nominal annual return, which isn't really very good for an investment portfolio in the long run (again, by historical standards, no comment on the future).

My boring portfolio has returned 9.2% nominal over the past decade, but there's a lot of drag in my portfolio from having a significant fraction of the portfolio dedicated to hedging against various risks, a totally vanilla old broad-based index fund portfolio would've had > 10% return over the same time period.

I think I've lived in a number of unusually hot housing markets and if I look at house price indexes the return has been 1.7x over the past decade for the hottest markets I've lived in that aren't SF over the past decade (Austin and Seattle), so the average is well below the "success" stories where people are bragging about doubling in a decade.

There are a few standard arguments people always bring up over the lunch table:

1. But I'm not throwing away money on rent

2. But the returns on my house are great because I'm heavily leveraged

3. But I can deduct the interest on my loan

(1) assumes you wouldn't have done anything else with the money. If, instead of buying a $1m house, you invest that in a boring portfolio that returned 10% a year over the past decade, you can pay rent and have plenty of money left over. $100k/yr in a rent is incredibly extravagant even in places where $1m can barely get you a house.

(2) assumes you couldn't be leveraged on your investments, but of course you could also take on debt to make investments. I don't do this because I don't want to take on the risk, but I don't think the risk is really different when it comes to owning a home, as many people I know found out when their mortgages ended up "underwater" after the 2008 crash.

(3) is fair, but the difference in returns is so large that this doesn't make up for it and it also assumes that you want a mortgage, which I don't, for reasons stated in (2).

At least in recent years, SF is a different story, but as someone working in tech, the value of a house in SF is highly correlated with my future earnings, which is the exact opposite of what I want in an investment. If the house prices continue to shoot upwards, it's relatively likely that the unvested stock I own in my employer will also shoot upwards. Even if my employer happens to do poorly, house prices going up in SF will very likely also mean that the tech industry is doing well, compensation continues to rise, it's very easy to find a lucrative job, etc.

I could see buying a house as consumption (it would be nice to have a place with no shared walls with neighbors, where I don't worry about playing loud music at 2am or hosting a party that runs late), but SF is the last place I'd want to do that.