If you are concerned about weighting too heavily into large cap, CRSP also has small and micro cap indexes. Take a look at their total market index methodology [1], which points you to the small and micro cap sister indexes. Even in FI/RE (financial independence/retire early) and Boglehead circles, there are a low percentage of people with $10M+ investable assets, the level at which tinkering with the kind of optimizations you are pointing out starts to become interesting. I picked CRSP because Vanguard uses them, but indexes like Wilshire's are about the same.
Most middle- to upper-middle class individuals with a 2+ decade investment time horizon (which should describe the majority of HN readership) are better served with starting on a relatively vanilla FIRE/Boglehead approach, and fine-tuning with allocations to broad bond/EM/International indexes as they go along and get a feel for their personal risk appetite.
The plentiful exceptions are people for whom investing is a hobby, they are good at tracking their own performance, and they have domain knowledge about a specific industry. Small-scale active investment can work, but for the vast majority of people who just want to set aside a comfortable retirement without having to spend a lot of time tinkering with it, I have yet to find a better generic approach than passive index tracking on very broad market indexes, though I'm certainly very open to suggestions.
I somewhat doubt passive indexing will ever catch on to the levels you mention due to human nature. Indexing only really works over extremely long time spans, and the data has so far consistently shown most people do not plan that far out for their personal finances. As long as most people are personally responsible for the majority of their retirement finances, I don't expect the consistently-applied delayed gratification a successful indexing strategy requires to cause "too many" people to sit in indexes. I classify that as a "would be a nice problem to have", so I'm not worried even if it somehow comes to pass.
Most middle- to upper-middle class individuals with a 2+ decade investment time horizon (which should describe the majority of HN readership) are better served with starting on a relatively vanilla FIRE/Boglehead approach, and fine-tuning with allocations to broad bond/EM/International indexes as they go along and get a feel for their personal risk appetite.
The plentiful exceptions are people for whom investing is a hobby, they are good at tracking their own performance, and they have domain knowledge about a specific industry. Small-scale active investment can work, but for the vast majority of people who just want to set aside a comfortable retirement without having to spend a lot of time tinkering with it, I have yet to find a better generic approach than passive index tracking on very broad market indexes, though I'm certainly very open to suggestions.
I somewhat doubt passive indexing will ever catch on to the levels you mention due to human nature. Indexing only really works over extremely long time spans, and the data has so far consistently shown most people do not plan that far out for their personal finances. As long as most people are personally responsible for the majority of their retirement finances, I don't expect the consistently-applied delayed gratification a successful indexing strategy requires to cause "too many" people to sit in indexes. I classify that as a "would be a nice problem to have", so I'm not worried even if it somehow comes to pass.
[1] http://www.crsp.com/files/Equity-Indexes-Methodology-Guide_0...