I wonder if index funds ever get widespread enough adoption that they start to drive prices of the major indexes up and underperform... maybe it sounds crazy right now since institutional investors don't really go for index funds heavily, but if there's a shift on those parameters I could see it happening.
Also, I've started to get more down on index funds as I look into them more. An index that has a weighted average of the stock market is always paying for past performance -- you're going heaviest on the largest market cap stocks always. In an era with both stability and upside for large market capitalization stocks it'll do well... in an era where the best private companies don't IPO until they've ran out most of their growth trajectory and then IPO once they've relatively stabilized, you're going to get hammered, no?
As you rightly realise, we can't have a situation where all money is blindly invested in passive index trackers, since everyone will be following the herd and no one will be leading it.
If active managers are under-performing, then at some point as passive funds grow we will reach an equilibrium point where they are causing such price distortion that they will produce opportunities for active managers to profit from mispricings, correcting the situation.
If you are concerned about weighting too heavily into large cap, CRSP also has small and micro cap indexes. Take a look at their total market index methodology [1], which points you to the small and micro cap sister indexes. Even in FI/RE (financial independence/retire early) and Boglehead circles, there are a low percentage of people with $10M+ investable assets, the level at which tinkering with the kind of optimizations you are pointing out starts to become interesting. I picked CRSP because Vanguard uses them, but indexes like Wilshire's are about the same.
Most middle- to upper-middle class individuals with a 2+ decade investment time horizon (which should describe the majority of HN readership) are better served with starting on a relatively vanilla FIRE/Boglehead approach, and fine-tuning with allocations to broad bond/EM/International indexes as they go along and get a feel for their personal risk appetite.
The plentiful exceptions are people for whom investing is a hobby, they are good at tracking their own performance, and they have domain knowledge about a specific industry. Small-scale active investment can work, but for the vast majority of people who just want to set aside a comfortable retirement without having to spend a lot of time tinkering with it, I have yet to find a better generic approach than passive index tracking on very broad market indexes, though I'm certainly very open to suggestions.
I somewhat doubt passive indexing will ever catch on to the levels you mention due to human nature. Indexing only really works over extremely long time spans, and the data has so far consistently shown most people do not plan that far out for their personal finances. As long as most people are personally responsible for the majority of their retirement finances, I don't expect the consistently-applied delayed gratification a successful indexing strategy requires to cause "too many" people to sit in indexes. I classify that as a "would be a nice problem to have", so I'm not worried even if it somehow comes to pass.
I wonder if index funds ever get widespread enough adoption that they start to drive prices of the major indexes up and underperform... maybe it sounds crazy right now since institutional investors don't really go for index funds heavily, but if there's a shift on those parameters I could see it happening.
Also, I've started to get more down on index funds as I look into them more. An index that has a weighted average of the stock market is always paying for past performance -- you're going heaviest on the largest market cap stocks always. In an era with both stability and upside for large market capitalization stocks it'll do well... in an era where the best private companies don't IPO until they've ran out most of their growth trajectory and then IPO once they've relatively stabilized, you're going to get hammered, no?