prompts:
    - "Solve in Python: {{ask}}"

  providers:
    - ollama:chat:codellama:7b
    - ollama:chat:codegemma:instruct

  tests:
    - vars:
        ask: function to return the nth number in fibonacci sequence
    - vars:
        ask: convert roman numeral to number
    # ...

[0] https://github.com/ollama/ollama

[1] https://github.com/promptfoo/promptfoo

https://huggingface.co/collections/google/codegemma-release-...

I am really liking the Gemma line of models. Thoroughly impressed with the 2B and 7B non-code optimized variants. The 2B especially packs a lot of punch. I reckon its quality must be at par with some older 7B models, and it runs blazing fast on Apple Silicon - even at 8 bit quantization.

Also looking fwd to use 2b models on iOS and Android (even if they will be heavy on the battery).

With models like CodeGemma and Command-R+ it makes more and more sense to run them locally.

  "llm.backend": "ollama",
  "llm.url": "http://localhost:11434/api/generate",
  "llm.modelId": "codegemma:2b-code-q8_0",
  "llm.configTemplate": "Custom",
  "llm.fillInTheMiddle.enabled": true,
  "llm.fillInTheMiddle.prefix": "<|fim_prefix|>",
  "llm.fillInTheMiddle.middle": "<|fim_middle|>",
  "llm.fillInTheMiddle.suffix": "<|fim_suffix|>",
  "llm.tokensToClear": ["<|fim_prefix|>", "<|fim_middle|>", "<|fim_suffix|>", "<|file_separator|>"],

I haven't found an open source VSCode or WebStorm addon yet that allows me to use a local model and implements code completion and commands as good as GitHub Copilot.

They either miss a chat feature and/or inline action / code completion and/or fill-in-the-middle models. And if they do, they don't provide the context as intelligently (? an assumption!) as GH's Copilot does.

One alternative I liked was Supermaven: It's really really fast and has a huge context window, so it knows almost your whole project. That was nice! But - one thing I ultimately didn't continue using it for: It doesn't support chat and/or inline commands (CTRL+I on VSCode's GH Copilot).

I feel like a really good Copilot alternative is definitely a still missing.

But: Regarding your question, I think GitHub Copilot's VSCode extension is the best - as of now. The WebStorm extension is sadly not as good, it misses the "inline command" function which IMHO is a must.

https://github.com/continuedev/continue

Blog post on repository context: https://tabby.tabbyml.com/blog/2023/10/16/repository-context...

(Disclaimer: I started this project)

Supermaven has a 300k token context. It doesn't seem like it has a ton of intelligence -- maybe comparable to copilot, maybe a bit less -- but it's much better at picking up data structures and code patterns from your code, and usually what I want is help autocompleting that sort of thing rather than writing an algorithm for me (which LLMs often get wrong anyway).

You can also pair it with a gpt4 / opus chat in Cursor, so you can get your slower but more intelligent chat along with the simpler but very fast, high context autocomplete.