The problem is that ARM is a licensable architecture, so there are lots of minor variations in sub-architectures.
The companies maintaining these have no motivation to consider the problem of maintaining /all/ of these, they're happy to basically copy/paste some other code until their CPU work.
Linus is saying that that's not going to work anymore, that they're going to have to come up with a way to consolidate their codebases.
That's probably the right solution, but it's a hard problem. NetBSD is probably the kernel that goes to the most effort to make clean, modular, orthogonal abstractions across hardware families and components, but it took them a lot of design effort up front, and continual policing of the abstraction boundaries and debugging of weird variants to make things actually work that way.
The companies maintaining these have no motivation to consider the problem of maintaining /all/ of these, they're happy to basically copy/paste some other code until their CPU work.
Linus is saying that that's not going to work anymore, that they're going to have to come up with a way to consolidate their codebases.