My idea is that you cannot have a huge machine that produces a whole wall in one go on a construction site of a single-family home, but you can have it on a factory. And the machine can work like 10x as fast as human builders.

Of course, to be economic, that machine should produce these walls day in day out without much interruption, hence the economy of scale.

Putting walls up is incredibly cheap, that's like the least interesting thing to automate. Plus it'd be hard (read: expensive) to move them from the factory to the site, and it'd be hard to get them into place when they're on site.

For things that are actually hard to build or easy to move, like cabinets or roof trusses, we're already building them in factories

Only if you are building the same wall all the time. As soon as some decides they want their house to not look like the one next door and thus makes some "trivial" changes your machine can't make the house anymore.

That is why you can't scale: people want their house to be unique.