You can run using your own OpenAI key by setting 2 environment variables:
(1) SKETCH_USE_REMOTE_LAMBDAPROMPT=False
(2) OPENAI_API_KEY=YOUR_API_KEY
To run entirely locally (using your own GPU and a model like Bloom) one would have to add a new prompt type to `lambdaprompt` (the package that this depends on), have a machine with enough GPU resources, and then add a slight modification to sketch.
Not sure if this is a business you're building out of this or an experiment. For real use for any of my customers, I would need to run this entirely locally.
I think it's really awesome though!
Curious what "enough GPU resources" looks like? Would a GeForce RTX 40 or 30 series card with 12-24GB of RAM be sufficient per user running locally on their machine?
You can run using your own OpenAI key by setting 2 environment variables: (1) SKETCH_USE_REMOTE_LAMBDAPROMPT=False (2) OPENAI_API_KEY=YOUR_API_KEY
To run entirely locally (using your own GPU and a model like Bloom) one would have to add a new prompt type to `lambdaprompt` (the package that this depends on), have a machine with enough GPU resources, and then add a slight modification to sketch.