What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project.
Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.
Jeff Geerling just did a video with a cluster of 4 Framework Desktop main boards. He put a decent amount of work into Exo and concluded it’s a VC Rugpull… abandoned as soon as it won some attention.
He also explored several other open source AI scale out libraries, and reported that they’re generally way less mature than tooling for traditional scientific cluster computing.
The founders of Exo ghosted the dev community and went closed-source. Nobody has heard from them. I wish people would stop recommending Exo (a tribute to their marketing) and check out GPUStack instead. Overall another rug pull by the devs as soon as they got traction.
Why can't that dev community just fork the project under a new name and maintain it properly? Picking up a third-party project is absolutely par for the course in FLOSS development.
Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.