Hacker News new | past | comments | ask | show | jobs | submit login

Koboldcpp (with ngrok if you need it) is another excellent self hosting solution.

13b will work on 16GB RAM, and 33b on 32GB RAM, with pretty much any dGPU for a little acceleration and RAM offloading.

Doubly so if you host it as an AI Horde node (so you have priority access to many models through the web browser).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: