That's true, Manhattan-style skycrapers aren't the only option. The transit-reachable parts of SF are already denser than some of what you mention as comparable, though: the Mission has 50% higher population density than West Hollywood.
One problem is that density is much lower in places like the Outer Sunset, in part because there's no cross-town subway. Total SF population could be increased significantly if the western half's population density could be brought up to what's already the case in the eastern half.
It's an easy problem to solve: allow more housing in the Outer Sunset. The demand for housing in SF is fierce, and throwing up some towers (or hell, just a-few-stories-tall buildings) will create a huge population boom.
Which will beget the subway.
As it is there is neither the political will, nor the practical population, to support such a thing.
One problem is that density is much lower in places like the Outer Sunset, in part because there's no cross-town subway. Total SF population could be increased significantly if the western half's population density could be brought up to what's already the case in the eastern half.