I'm on team open source. To me the exciting thing was ollama downloading the 7B and running it on a 5yo cheap lonovo and getting a token rate similar to the first release of ChatGPT.
Running local on CPU opens so much possibilities for smart and privacy focused home devices that serve you.
In my test it hallucinated confidently but my interest is in simple second brain like rag. "Hey thingy, what is my schedule today?"
Need it to be a bit faster though as the thinking part adds a lot of latency.
The thinking is quite fascinating though, I love reading it. Especially when it notices something must be wrong. It will probably be very helpful to refine answer for itself and other models.
It does add latency of course, but I still think that I could provide all AI needs of my company (industrial production) with a simple older off the shelf PC. My GPU is decently recent, but the smallest model of the series and otherwise the machine is a rusty bucket.
I didn't test it thoroughly yet, but I have some invoices where I need to extract info and it did a perfect job until now. But I don't think there is any LLM yet that can do that without someone checking the output.
Running local on CPU opens so much possibilities for smart and privacy focused home devices that serve you.
In my test it hallucinated confidently but my interest is in simple second brain like rag. "Hey thingy, what is my schedule today?"
Need it to be a bit faster though as the thinking part adds a lot of latency.