Transportation. The bulk of the economy is still on the west side of the Bay, and crossing the Bay can be a huge pain in the ass. BART is frequently unreliable, expensive, infrequent, and stops early.

On top of that the transportation on the Oakland side is also highly lacking, further increasing the friction for people working in SF but living in the East Bay.

On top of that the transportation on the SF side is also highly lacking. If you work within walking distance of a BART stop, power to you. For everyone else it means catching a transfer to the worst transit agency I've ever seen: MUNI.

In other words, the only commute that really competes with living in the city proper is one where you're living walking distance to a BART stop, and you work within walking distance to a BART stop at the other end. Any other use case becomes an irredeemable mess, substantially limiting the attractiveness for residents.

Transportation has, and is, and will continue to be, the Achille's Heel of San Francisco and its surroundings. The continuing utter failure to invest in transit infrastructure will be the city's undoing.