Don't underestimate the value of batching even for personal use. You can get MUCH better results from a language model if you sample a couple outputs and choose the best to continue.
This kind of usage isn't especially economical for hosted use-- but for personal use it would mostly be using idle resources and you can get extra samples almost for free.
A bunch of people getting multiple completions and choosing which one they'd prefer to continue might make for some really useful training data too.
This kind of usage isn't especially economical for hosted use-- but for personal use it would mostly be using idle resources and you can get extra samples almost for free.
A bunch of people getting multiple completions and choosing which one they'd prefer to continue might make for some really useful training data too.