To execute this automatically in an efficient way we need a query planner which can look at the whole pipeline and decide to use a different set of primitives if the naive/implied ones aren't sufficient. What you're talking about is implemented in relation database management systems but it requires the query planner to know about the whole system - that is it's the opposite of Unix - there is a piece of the system that needs to know about the whole system.
As far as Unix is concerned, the whole list is not stored in memory. Pipes are buffered streams (they are iterators, just with a buffer attached, which makes them more efficient not less).
> What you're talking about is implemented in relation database management systems but it requires the query planner to know about the whole system - that is it's the opposite of Unix
And that's exactly my point: that this line of thinking about how to make these things right will lead to overcomplicated, bloated system.
As far as Unix is concerned, the whole list is not stored in memory. Pipes are buffered streams (they are iterators, just with a buffer attached, which makes them more efficient not less).