I don't trust the code quality evalution. The other day at work I wanted to split my string by ; but only if it's not within single quotes (think about splitting many SQL statements). I explicitly asked for stdlib python solution and preferrably avoid counting quotes since that's a bit verbose.
GPT4 gave me a regex found on https://stackoverflow.com/a/2787979 (without "), explained it to me and then it successfully added all the necessary unit tests and they passed - I commited all of that to the repo and moved on.
I couldn't get 70B to answer this question even with multiple nudges.
Every time I try something non GPT-4 I always go back - it's feels like a waste of time otherwise. A bit sad that LLMs follow the typical winner-takes-it-all tech curve. However if you could ask the smartest guy in the room your question every time, why wouldn't you?
Thanks for the feedback, could you please post the cached Phind link so we can take a look?
It might also be helpful to try Phind Chat mode in cases like this.
EDIT: It seems like Phind-70B is capable of getting the right regex nearly every time when Chat mode is used or search results are disabled. It seems that the search results are polluting the answer for this example, we'll look into how to fix it.
I've tried it with a question which requires deeper expertise – "What is a good technique for device authentication in the context of IoT?" – and the Search mode is also worse than the Chat mode:
The search was heavily diluted by authentication methods that don't make any sense for machine-to-machine authentication, like multi-factor or biometric authentication, as well as the advice to combine several methods. It also falls into the, admittedly common, trap of assuming that certificate based authentication is more difficult to implement than symmetric key (i.e. pre-shared key) authentication.
The chat answer is not perfect, but the signal-to-noise ratio is much better. The multi-factor authentication advice is again present, but it's the only major error, and it also adds relevant side-topics that point in the right direction (secure credential storage, secure boot, logging of auth attempts). The Python example is cute, but completely useless, though (Python for embedded devices is rare and in any case you wouldn't want a raw TLS socket, but use it in a MQTTS / HTTPS / CoAP+DTLS stack, and last but not least, it provides a server instead of client, even though IoT devices mostly communicate outbound).
You're right! It solved it. I didn't know about the Code/Search distinction. I still struggled for it to write me the unit tests. It does write them, they just don't pass. But this is definitely much closer to GPT4 than I originally thought.
I didn't take a look at the code, but to me it sounds quite dangerous to take an implementation AND the unit tests straight from an LLM, commit and move on.
It's very powerful, I can enter implementations for any algorithm by typing 5 words and clicking tab. If I want the AI to use a hashmap to solve my problem in O(n), I just say that. If I need to rewrite a bunch of poorly written code to get rid of dead code, add constants, etc I do that. If I need to convert files between languages or formats, I do that. I have to do a lot more code review than before, and a lot less writing. It saves a huge amount of time, it's pretty easy to measure. Personally, the order of consultation is Github Copilot -> GPT4 -> Grimoire -> Me. If it's going to me, there is a high probability that I'm trying to do too many things at once in an over-complicated function. That or I'm using a relatively niche library and the AI doesn't know the methods.
I take it to mean that the code quality deserves more scrutiny because you can't guarantee what it has provided is quality code, without reviewing it first.
The same applies to brand new devs — it's normal to apply a little more scrutiny because they simply don't have the experience to make the right decisions as confidently (or frequently) as someone more senior.
It's an analogy and the natural fact that output reflects experience and practice over time.
Reminds me of a Facebook thread I saw a few days ago, on the topic of 3D printing houses. All the comments were angry dismissive "hurr durr that's clearly poor quality work" with no further justification of their position, and it struck me how similar the overall energy was to the "all AI image generation is bad and shit and is also heinous immoral theft and you're literally the worst person in the world and yous should feel bad" sort of raging that you see any time someone posts some SD or Midjourney or whatever pic of a cute puppy riding a tricycle. These comments originate from people who've spent their lives learning skills that are now largely replaceable by a few gigs of download and a Python tutorial. No wonder they're upset.
Right...because requiring developers to actually grasp the code they're outputting is gatekeeping.
If you want to pretend the rush to AI won't lead to more incompetent chefs in the kitchen than we already have (which is too many as it stands) then feel free, but acting like it's some kind of "party" people are being kept out of is daft.
Standards exist for a reason, not just to make people feel bad for not meeting them.
You’ve still got to avoid prompting for questionable code in the first place, eg, splitting SQL statements on semicolons with an ad-hoc regex is going to fail in edge cases, but may be sufficient for a specific task.
Yes more than sufficient for an internal tool - we can assume good intentions of the users of the tool since people want for this to actually work and have no intention of hacking.
Except now it's a vector if anyone gets access to this internal tool.
I would be fine with this for one off scripts but absolutely can not consider anything less than full sql parsing or something equally robust if it is exposed over the network, even if only internally and behind authn and authz.
For this reason, I tend to ask LLMs additional questions like: "show me another way to do this" or specifically "how would someone with a higher need for security write this?"... knowing that I'm likely to get a more refined answer from different sources that have probably discussed deeper security implications around the same goals, for instance.
Right on. These days my llm-assisted workflow feels very similar to the 20% of my day that I used to devote to code review, just now it’s more like 60% of my day.
I’m finding it’s more effective (and pleasurable) to write using GitHub CoPilot and CMD-RIGHT (accept next word). I put a detailed doc comment above and write in tandem with copilot. I’ve written the structure and I review as I write jointly with the model.
This way I don’t need to review a block of code I didn’t write.
<aside>I had an experience yesterday where CoPilot correctly freed all the memory in correct order at the end of a rather complicated C algorithm, even where there was nested mallocs.</aside>
Yeap, I want a code-review bot that just says "this is very improbable; are you sure you didn't mean x instead?"
The old Coverity used to achieve similar results in a different way, spotting probable mistakes based on patterns its heuristics found in the rest of the same codebase.
Hopefully not, I feel it's a waste of time. The time spent on stupid minor mistakes by github copilot I didn't catch probably doesn't really compare to the time I would've spent typing on my own. (I only use that stuff for fancy code completion, nothing more. Every LLM is absolutely moronic. Yesterday I asked chatgpt to convert gohtml to templ, to no avail ...)
Agreed, though i'm _really_ interested in trying 1M token Gemini. The idea of uploading my full codebase for code assist stuff sounds really interesting. If i can ever get access to the damn thing...
I'm curious how they'll handle this. My understanding is that it takes quite a long time to get an answer, since there's no magic "semantic database" built for you behind the scenes.
That use-case seems inefficient to solve like that in the long run as well, like if you really would have to use a million tokens to do every small query you require on your data it would be prohibitively costly except doing as an experiment.
Gemini is much better than the free version of GPT 3.5 though. At least in my experience.
Microsoft’s enterprise co-pilot is also fairly decent. It’s really good at providing help to Microsoft related issues or helping you find the right parts of their ridiculously massive documentation site. Which probably isn’t too weird considering.
"zsh rename index.html.1 to image_1.png for many images"
Gemini
Here's a zsh script that will rename multiple files named index.html.1 to
image_1.png in your current directory and its subdirectories:
Bash
#!/bin/zsh
# Loop through all files in the current directory and subdirectories
for file in **/index.html.1; do
# Extract the base filename and extension
base_name="${file%.*}"
extension="${file##*.}"
# Rename the file with the new name and extension
mv "$file" "${base_name}_1.png"
# Print a message for each renamed file
echo "Renamed $file to ${base_name}_1.png"
done
ChatGPT3.5
To rename multiple files in zsh, you can use the zmv function. Assuming you
want to rename files from index.html.1 to image_1.png for many images, you
can create a pattern to match the filenames and then define the new names
using capturing groups. Here's an example:
zsh
autoload -U zmv
zmv -n '(*).html.(*)' 'image_$1_$2.png'
Not a great first impression of Gemini. ChatGPTs answer isn't perfect but its a lot closer to correct, only needing me to remove the extra 'index' capture of $1.
Curious if someone could commit some light copyright infringement and post what GPT4 says to the same prompt.
Edit: Also Phind-34B probably gives the best answer, with the correct capture.
To rename multiple files in zsh from index.html.1 to image_1.png for many
images, you can use the zmv command, which is a powerful batch renaming tool
available in zsh. If you encounter the error zsh: command not found: zmv, you
can load it using autoload -U zmv 1.
First, ensure zmv is available by adding autoload -U zmv to your ~/.zshrc file
or run it directly in your shell session 2.
Here's how you can use zmv to rename your files:
zmv -n 'index.html.(*)' 'image_$1.png'
Here's what gpt4-turbo-preview outputs (with max output of 256 tokens, so the result was truncated).
P.S. have you tried testing what happens when you clearly describe what you want? The prompt you're using is really low quality - more like a google search. If you asked me a question like that I'd tell you to clearly explain what it is you want.
If you have multiple files with names like `index.html.1`, `index.html.2`, etc., and you want to rename them to `image_1.png`, `image_2.png`, and so on, using Zsh, you can accomplish this with a simple loop and the `mv` command.
Here is how you could do it in a Zsh terminal:
```zsh
autoload -U zmv
zmv -W 'index.html.*' 'image_*.png'
```
The `autoload -U zmv` command loads the `zmv` function, which is a powerful utility in Zsh for batch renaming files. The `-W` option tells `zmv` to use wildcard patterns for matching and renaming files.
In the `zmv` command:
- `'index.html.*'` is the pattern to match. It matches any file that starts with `index.html.` followed by any characters.
- `'image_*.png'` is the replacement pattern. The asterisk `*` in the replacement pattern corresponds to the `*` in the match pattern, so each matched number after `index.html.` gets placed where the `*` is in `image_*.png`.
**Important:** Always make sure
In stable diffusion we build x/y plots to evaluate the results due to seed variance. I find it interesting that LLM guys (seemingly) never do that, since their answers shouldn't be deterministic too.
In my experience, Bing's image search is way better than Google's. Also, I'm not going to use a search engine that I have to log in or do a captcha for.
The time complexity for all matching a string against any fixed regular expression is O(length of string).
If you want to talk about constant factors, we need to leave our comfortable armchairs and actually benchmark.
[Just to be clear, I am talking about real regular expressions, not Franken-xpressions with back-references etc here. But what the original commenter described is well within the realm of what you can do with regular expressions.]
You are right about escaped quotes etc. That's part of why parsing with regular expressions is hard.
The time complexity for deciding whether an N-letter string matches a regex or not, is O(N). The time complexity of finding all matches is not O(N) - which is needed in OPs case, because they want to split the string.
Also, OP's solution uses lookahead assertions, so it's not a real regular expression.
(I wonder if we can summon @burntsushi for expert opinion on this?)
You are right that the lookahead might be expensive.
(There's probably a way that a sufficiently smart compiler of (ir-)regular expressions can optimize this expression to be still matchable quickly; but Python's regular expression matcher is probably not that smart. I'm not sure if any real world matcher is.)
> The time complexity of finding all matches is not O(N) [...]
If you are happy find a maximal set of non-overlapping matches, you can still do it in O(N). By 'maximal' I mean that you can't greedily find another match (without removing any of the existing matches.)
A sketch of the technique is: take your pattern and wrap it up like this '.?{pattern.?}' (where ? means non-greedy repetition) and match that against your input string. You can do non-greedy repetition and the very limited form of sub-pattern capturing that you need to find all the matches of 'pattern' without breaking O(N) time.
I'm not sure whether you can find the global maximum number of non-overlapping matches, instead of a just a greedy maximum, in O(N) time.
"Can you give me an approach for a pathfinding algorithm on a 2D grid that will try to get me from point A to point B while staying under a maximum COST argument, and avoid going into tiles that are on fire, except if no other path is available under the maximum cost?"
I've never found an AI that could solve this, because there's a lot of literature online about A* and tiles with cost, and solving this requires a different approach
I don't care much for benchmarks, many models seems to be contaminated just to approach proprietary models in coding benchmarks.
I had never tried Phind before, but gave Phind-70B a spin today and so far found it to be really good for coding writing and understanding, maybe even GPT-4 level. Hard to tell for sure since I only tested it on a single problem: Writing some web3 code in typescript. This is what I did:
- Gave it some specifications of a react hook that subscribes to a smart contract event and fetches historical events starting from a block number. It completed successfully.
- Took this code and gave it to GPT-4 to explain what it did, as well as finding potential issues. GPT gave a list of potential issues and how to address.
- Then I went back to the Phind and asked it to find potential issues in the code it had just written, and it found more or less the same issues GPT-4 had found.
- Went back to GPT-4 and asked to write a different version of the hook.
- Took the GPT-4 written code and asked it to explain the code, which it did successfully (though I think it lacked more details than the GPT-4 explanation of the code written by Phind).
I will be testing this more over the next days. If this proves to be in the GPT-4 ballpark and the 70b weights are released, I will definitely replace my ChatGPT plus subscription with Phind Pro.
Not an expert at all. But just wanted to let the creators know: I've been using Phind almost daily for some months now and it's been awesome. Whenever I accidentally do a web search I recognize what a game changer this is. (ChatGPT probably as well, but never used it.) Last week I was under pressure at work and I used it for stuff like: "How can i capture output from a command and print it line by line to the console with Rust", and must say that kind of time and energy savings are very significant.
Just wait for people to stop using SO, at which point the LLMs won't have a high quality training set for new questions, so you won't get good answers from the LLMs anymore...
In what world are you living in? That's maybe true in noob land. Literally all the problems I have are being solved in github issues, if at all. When has documentation been 90% sufficient for anything? In the 80s?
/e: sorry, sounds a bit stand off-ish.
Let me give an example: I was trying to find a way to clone a gorm query to keep the code clean. The documentation doesn't have anything (no, .Session isn't a solution) and the only place I had was issues discussing that. Apparently you can't. So I'll be ditching gorm and move to pgx in the near future. That's how it happens for me all the time. The documentation is lacking the hard part, always.
The LLMs will read the actual source code which is way better than the documentation (as any iOS engineer will tell you). For private codebases the companies can provide custom-trained LLMs. Techniques like "Representation Engineering" will at some point also prevent against accidental leakage of private codebase source code.
Depends on the language, but many things happen on Discord now (which is very annoying since it's not indexable by search engine and you need to ask the question to get the answer…)
I’ve been doing this in my private codebase. When copilot hallucinates a function, I just go and write the thing. It’s usually a good idea, and it will re-hallucinate the same function independently in another file.
The only way this is useful in the context of code is if:
* The LLMs have a sufficient "understanding" of the request and of how to write code to fulfill the request
* Have a way to validate the suggestion by actually executing the code (at least during training) and inspecting the output
From what I've seen we are still far away from that, Copilot and GPT-4 seem heavily reliant on very well-commented code and on sources like Stackoverflow
If I were training a code model I'd take a snippet of code, have the existing LLM explain it. Then use the explanation and the snippet for the test data.
I don't use LLMs a lot, maybe once a week or so. But I always pick Phind as my first choice because it's not behind a login and I can use it without giving my phone number. Hopefully you'll keep it that way!
I think there’s room in the market to subsidize real users. Phind delivers absurd value, so I think the majority of paying users could account for the tech-averse or privacy-conscious
Phindational models, phintech, Phinterest, phinder… it might be the best startup name of all time. Hell, startup a password manager and call it Phinders’ Keeper.
Very nice. I've been working with GPT4 since it released, and I tried some of my coding tasks from today with Phind-70B. The speed, conciseness, and accuracy are very impressive. Subjectively, the answers it gives just feel better than GPT4, I'm definitely gonna give pro a try this month.
I prefer Phind's web search with LLM to both Google search and GPT-4. I have switched my default search engine, only using Google for finding sites, not for finding information anymore.
GPT-4 might be a better LLM but its search capability is worse, sometimes sends really stupid search keywords that are clearly not good enough.
I won’t steal phind’s thunder but kagi is another great modern tool to have, and much more reliable than google for a technical user IMO. Obviously Phind is irreplaceable for complex or chat-based technical questions, but Kagi sees much more use from me daily for syntax stuff, Wikipedia searches, finding and relating papers, etc.
I tried asking "What is the size of Phind-70B's context window?" and it couldn't answer the question. Strangely, it immediately found the page with the answer (https://www.phind.com/blog/introducing-phind-70b) but refused to acknowledge that the answer was there. I tried asking several ways. It even quoted the exact answer in the displayed snippet, but still said there was no answer!
Since you're here: have you considered moving to other, better generalist base models in the future? Particularly Deepseek or Mixtrals. Natural language foundation is important for reasoning. Codellama is very much a compromise, it has lost some NLP abilities from continued pretraining on code.
I tried a question about Snobol4 and was impressed with what it said (it couldn't provide an exact example due to paucity of examples). When testing more mainstream languages I have found it very helpful.
Hello Michael, lovely to see this, congrats. Do you already have an API? I could not see it on the site. If not, then do you know around when we can expect it? I am building a desktop BI app with hosted and local LLMs (need schema inference and text to SQL). Would be nice to have Phind as an option for users. Thanks
I'd suggest logging in in that case -- you will still get your free uses. The Phind-70B counter for non-logged in users has carried over from when we offered GPT-4 uses without a login. If you've already consumed those uses, you'll need to log in to use Phind-70B.
I have been using Phind almost daily for the past 3-4 weeks and the code it produces is pretty good and it is runnable on the first try more often compared to ChatGPT. Most of the time the answer is somewhat accurate and points me in the right direction.
ChatGPT (with GPT 4) has been slow af for me for the past 2+ months but I like studying a topic using ChatGPT, it is more verbose and explanatory when explaining things to you.
Maybe a purpose-built dedicated AI model is the right path. A model that does well in fixing bugs, writing feature code, and producing accurate code will not be a good tool for or conversational studying. And vice versa.
Also, I don't like that Phind is not handling the follow-up question that well when there are multiple kinds of questions within the same thread. ChatGPT is good at this.
I haven't actually because Phind is working for me so far whenever I have code-related questions or when I need to refactor my code. TIL that I can customize the answer style preference, will give it a try!
I'm impressed with the speed, really impressed, but not so much with the quality of the responses. This is a prompt I usually try with new LLMs:
> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.
GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind in the other hand, in the two or three times I tried, ignores the POST part and focus on GET only.
Maybe that problem is just too hard for LLMs? Or the prompt sucks? I'll see how it handle other things since I still have a few tries left.
Does anyone outside the Go community call it a "RoundTripper"? I know what a retry is (and things like exponential backoff) and what GET and POST are, but not that, but I also hate Go, so...
EDIT: ah, followup replies elucidated me, it's just a goofy name for a Go-only thing
Thanks, can you send the cached link please? I'd also suggest trying Chat mode for questions like this, where there are unlikely to benefit from an internet search.
Just tried your query now and it seemed to work well -- what are your thoughts?
Thanks for the links. It seems like it switched to Phind-34B, which is worse.
Phind-70B seems to be able to get the right interface every time. Please make sure that it says Phind-70B at the top of the page while it's generating.
I'm not sure what you mean that it "forgot" about POST? Even as an experienced Go developer, I looked at the code and thought it would probably work for both GET and POST. I couldn't easily see a problem, yet I had not forgotten about POST being part of the request. It's just not an obvious problem. This is absolutely what I would classify as a "brain teaser". It's a type of problem that makes an interviewer feel clever, but it's not great for actually evaluating candidates.
Only on running the code did I realize that it wasn't doing anything to handle the problem of the request body, where it works on the first attempt, but the ReadCloser is empty on subsequent attempts. It looks like Phind-70B corrected this issue once it was pointed out.
I've seen GPT-4 make plenty of small mistakes when generating code, so being iterative seems normal, even if GPT-4 might have this one specific brain teaser completely memorized.
I am not at the point where I expect any LLM to blindly generate perfect code every time, but if it can usually correct issues with feedback from an error message, then that's still quite good.
This isn't a brain teaser at all. It's a direct test of domain knowledge/experience.
There are countless well-documented RoundTripper implementations that handle this case correctly.
This is the sort of thing you whip up in three minutes and move along. To me it seems like a perfect test of LLMs. I don't need an injection of something that's worse than stackoverflow polluting the code I work on.
A fun little challenge I like to give LLMs is to ask some basic logic puzzles, i.e. how can I measure 2 liters using a 3 liter and a 5 liter container? Usually if it's possible, they seem to do ok. When it's not possible, they produce a variety of wacky results. Phind-34B is rather amusing, and seems to get stuck in a loop: https://www.phind.com/agent?cache=clsxpravk0001la081cc9dl45
1. phind was by far the best - gave me solution in just 2 steps
2. Grok was second best - it did arrive at the solution but with additional non-sense step. But the solution was correct.
3. To my surprise GPT-4 could not solve the prompt and in fact gave a wrong answer in 4 steps - "Now you should have exactly 4 liters in the 5-liter container." which is not what I asked
4. As expected Gemini pro was the worst. It asks me to pour completely filled up 3L container into 5L and then you will be left with 2L in 3L container.. LOL that does not even make sense.
These are interesting tests. I wonder how far we are away from AIs solving these (the ones that have no solution) without any special programming to teach them how.
Do you have an API that could be plugged into https://aider.chat/ ? It's by far the best way to use GPT4 for coding, in my experience, and more speed is exactly what it could use. But it needs an OpenAI compatible API.
Aider has been great! Really looking forward to seeing a phind and even Gemini 1.5 plugin eventually. Def been a lovely improvement to my workflow. I've been keeping a close eye on Mentat as well but haven't yet tried it.
That's extremely disappointing. They have time to build a Visual Studio Extension that competes with Cursor, but don't have time to release an API that would enable hundreds of new extensions/workflows.
Only reason I pay for ChatGPT Plus is because they have an API and I'm building products off of their API. I use Phind more for work, but I'm not going to pay anything unless they have an API.
Weirdly enough, when I asked "give me a formula for the fourier transform in the continuous domain" to the 70B model, it gave me a latex-like formatted string, while when asked for "give me pseudocode for the fft" I got a nice code snippet with proper formatting. The formulas though were both correct. We're not at Groq level of speed here, but I have to say, it looks pretty good to me.
cache=uyem9mo96tjeibaeljm1ztts for the devs if they wanna look it up.
I understand why they’re doing this from a cost and dependency perspective, but I’ve pretty much stopped using Phind since they switched over to their own models. I used to use it in the past for thing like API docs summarization, but it seems to give mostly wrong answers for that now. I think this is mostly a “RAG doesn’t work very well without a very strong general model parsing the context” problem, which their prior use of GPT-4 was eliding.
I used it for awhile and it was pretty good at Bash or Emacs Lisp one-liners but it was wrong often enough that it was faster to just search on Kagi for the information that I want first, instead of performing N searches to check the answer from Phind after querying Phind.
Phind founder here. Thanks for the feedback -- I'd love to hear your thoughts on this new model. You can try it for free, without a login, by selecting it from the homepage: https://phind.com.
I just tried using the 70B model and the answer was listed as being returned using the 34B model instead of the 70B model and was wrong. Is there some logic that ignores user choice, depending on what the service thinks can be answered?
I tried the model and asked it to write a kubernetes operator with required DockerFiles, Resources, application code.. Asked it to migrate application to different languages. It looks like it's pretty capable and fast. It is impressive.
As someone who has utilized Nvidia Triton Inference Server for years it's really interesting to see people publicly disclosing use of TensorRT-LLM (almost certainly in conjunction with Triton).
Up until TensorRT-LLM Triton had been kind of an in-group secret amongst high scale inference providers. Now you can readily find announcements, press releases, etc of Triton (TensorRT-LLM) usage from the likes of Mistral, Phind, Cloudflare, Amazon, etc.
Phind makes impressive claims. They also claimed that their fine tune of codellama beat gpt4, but their finetune is miles behind gpt4 in open domain code generation.
Not impressed. Also this is a closed walled garden model.
We noticed that the training run crashed because one of the GPUs fell off the bus. Power cycling the host server didn't help and diagnostics showed thermal damage. We were able to swap in a different node, but apparently the entire host server needed to be replaced.
We've generally noticed a relatively high failure rate for H100 hardware and I'm not quite sure what is behind that.
Is there any generalizable measure of how any of these models (or their client implementation) handle code(base) context that's sent along each editing request?
For my use cases this seems to be as crucial a measure as the general coding responses per file / selection / request and where implementations like Cody[0], Cursor.sh[1] or aider.chat[2] stand out
I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak.
If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.
After running a bunch of models on my own PC (a pretty good one), I have to say by FAR the best results for coding has been with Deepseek models. However, I just spent 20 minutes playing with this Phind 70B model and it's totally nailing the questions I'm asking it. Pretty impressed.
- Hybrid offloading with llama.cpp, but with slow inference.
- Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context.
30B-34B is more of a sweetspot for 24GB of VRAM.
If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.
I’m curious how you find the Swedish from different models. GPT-4 seems to return perfectly grammatical Swedish but a Swede friend says it reads like English. Do you notice this?
I’d love to have models that are better at idiomatic usage of other languages, so they can generate language learning content.
Terrific stuff. I always enjoy using Phind for dev related questions.
Is it possible the chat history gets some product love? I would like to organize my conversations with tags, and folders. Make it easier to go back to what was said in the past instead of asking the question again.
Yup, that works (10 uses avail). Though i wasn't too concerned with actually using it, just thought it was interesting and wanted to expose that maybe-bug.
Can we get a few accessibility fixed? The expandable button after the sign in button and the button after that are unlabeled. The image on the heading at level 1 has no Alt-text. The three buttons after the "Phind-34B" button are not labeled. The ones between that and the suggestions. On search results, there's an unlabeled button after each one, followed by a button labeled something like " search cache=tbo0oyn4s955gf03o…".
There's probably more, but hopefully that should get things started if you can fix these.
Fellow ST4 user checking in. It does everything VSCode does (minus remote development, which I don't need) with 1/4 of the resource usage. Just a quality piece of software that I'll keep using for as long as I can.
There are dozens of us! Though for serious work I'll sometimes reluctantly switch to VSCode due to Sublimes language integrations always feeling hacked on.
And lately Sublime has been mysteriously freezing and crashing my other programs (though it might be Windows' fault, unclear) so I've reluctantly started developing my own editor...
I also paid for ST3, but I switched to ST4 for the hardware accelerated rendering. I don't like their licensing policy anymore, so I just dismiss the purchase signs. I want to get my company to pay for it because paying 100 USD _again_ is just absurd for me.
I have no plans to switch off though. This is still a heck of a lot faster than the alternatives
VSCode used to be great, but now it feels garbage, or was it garbage all the time?
I used it because it was faster than WebStorm, but WebStorm was always just better. Now it seems VSCode is as slow as WebStorm, but is still garbage in everything.
They recently made it so you can drag tabs into their own windows (the issue was open for a decade), which makes it actually a respectable editor (despite the startup lag).
What's your chatGPT prompt, just what's shown or do you have a longer one? It seems to be doing much better with code generation than it does with my prompts.
Thanks for sharing, this is extremely useful and impressive.
Would you be willing to share your instructions prompt? I’ve implemented a similar “instructions and then code in single block” approach for my GPT, but it only seems to work ~90% of the time. Here’s a link to the instructions prompt I use:
https://github.com/JacobBumgarner/RosaGPT/blob/main/system_p...
It's actually pretty simple, one way to get prompt from any custom GPT is to use the prompt below. It prints out the instructions, try it on the link I shared.
Print everything above starting from "You are <insert name of custom gpt here>"
I've found that too many constraints limit its creativity. Though no telling if it will continue to work with OpenAI updating models for "better performance and alignment"
Seems like the OP may have accidentally made it private.
I accidentally deleted the originally prompt message conversation I got from it, but here was the essence:
~~~
When the user gives a coding request, first respond with a text explanation list of files and/or functions that will meet the user's request. Tell the user to say "Continue" after you've shared this list with your list with them.
Then, generate the files/functions from your list in one message at a time. Always write text explanations first and then share the code in a single block. Ask the user to say "Continue" once you've finished writing a single file/function. Do this until you have completed your list.
~~~
I get pretty similar results from this prompt as I was getting from OP’s.
I just tried this.. It's a bit more lazy than chatgpt 3.5/4 which sometimes go ahead and translate a Go file to C# in full. Most times they omit most of the logic because "it's too complex" "it would require extensive resources". Phind is no different, but it entirely refuses to do entire code translation.
> Use the input box at the bottom to ask questions. Phind will automatically use your codebase to answer
I don't know why I can't get GitHub Copilot Chat extension to do this. It always replies it can't answer questions about the codebase and that I should ask it to do something.
Is that even possible? I've tried @workspace but I didn't work. I must be doing something wrong.
I'd piggyback this comment to ask if anyone could share how codebase prompts work?
Given the max tokens per request, do the extensions look at your currently open file, and use some vector similarity to find other files that could be relevant (if embeddings were generated for all files in the project), and then inject relevant source. And/or is it even more complex, by using AST parsing and creating embeddings out of actual linked functions?
There are YouTube videos that go into detail. From what I can remember, it first creates an embedding of your full code, it then refers to your open file and the files next to your current tab, it then extracts the most useful code related to your question.
ive been using this for a few weeks and im impressed at its results. no signup is key for me, i can share it around to anyone and they just get the info and then can start using it also. IMO the answers are fine for my tasks as a a substiute for a junior dev. its slightly faster then searching google cause it saves me a few clicks - handy if your already screen sharing with someone and just looking to fact check.
I was considering signing up for the pro plan. Now I won’t even give them my email. I tried the model and it is genuinely nice, but this is a huge red flag.
Awesome! I’ve been using phind a little over a year now since it was originally posted on HN. I prefer it over gpt. I’ve run into some weird issues where answers just loop or repeat after really long question threads. I can’t recall model that was being used but I’ll try and find some cached links I can share!
Every day now there are new AI models especially LLMs, which might warrant some consideration from a wide part of the human population. In a couple years we will have multiple new announcements per hour and we might need some earlier models to evaluate these new developments and test them. For Phind-70B in particular, I hope that lmsys will share a version that will be part of the human evaluation leaderboard so we get a rounded evaluation. But for code assistants there should be a totally separate impartial evaluation benchmark, ideally still human judged for another year or so but eventually maybe some way of having the models fighting out competitive coding battles that they can help create.
> In a couple years we will have multiple new announcements per hour
Models are research output. If 10 new models are being announced every day in a couple years, it would mean that generative AI research has failed to stabilize and produce a stable, reliable component ready for product engineering. And if that's where we are in a couple years, that's almost certainly a sign that the hype was misplaced and that money is chasing after itself trying to recoup sunk costs. That's a failure scenario for this technology, not what an AI-optimist (you otherwise seem to be one) should be anticipating.
I referee for a lot of the top machine learning conferences and yes I am very optimistic about AI and its impact on humanity. The amount of exciting new papers in machine learning and AI was definitely on an exponential rise for a decade since about 2012 or so, and the total production has kept increasing even during the last couple of years when the submissions in some top annual conferences exceeded 10k. Not every paper results in a useable model but a higher fraction of papers come with code and pretrained weights over time. Many of these papers will never be read by many more than the reviewers and the group who wrote them and a couple friends, but it does not speak necessarily to the quality of the work itself or the potential impact it could have on every possible future if we found better ways to separate the useful information. As the exponential increase in total compute becomes more widely accessible there are exponentially more applications that are of broader interest and will have even bigger impact than nowadays. I don’t think that the model of reviewing 10s or 100s of thousands of papers in conferences, or playing the popularity contest on social media is going to be productive so we need better methods for advancing the useful ideas more quickly. (Case in point: the mamba state space model by Gu and Dao was rejected from a conference this winter, but it happened to be advertised enough at a keynote presentation by Chris Re with a packed audience at neurIPS23, so the model was picked up by a lot of people who used it and submitted applications that used it to the ICML conference already.) I also don’t think that some of the biggest companies have enough manpower, motivation and interest in going alone, though of course they can easily stay ahead of the game in specialized areas with their own resources.
That’s not true. Both good science and market-driven engineering favor continued iterations on existing ideas looking for improvements or alternatives. We’re often exploring a giant space of solutions.
Unlike many fields, the A.I. people are publicly posting many of their steps in this journey, their iterations, for review. While it brings lots of fluff, such openness dramatically increases innovation rate compared to fields where you only see results once or twice a year. Both people using cloud API’s and FOSS developers are steadily increasing effectiveness in both experimentation and product development. So, it’s working.
That doesn't follow at all. It just means that there are still low-hanging fruits to pursue for better (smarter, faster, larger context etc) new models, but it doesn't say anything about the stability and usefulness of existing models.
> LLMs are more like apps being produced by different companies trying to capture walled gardens, and their open source counterparts.
I think the analogy to the web is stronger than that.
For now the LLMs are mostly separate, but it won't be long before LLMs emerge that make API calls to other LLMs, sometimes over the internet.
In due course, expect meta-LLMs to emerge that aggregate knowledge from other LLMs by talking to them, rather than by training on their data. Those meta-LLMs which optimise for competitive quality results will have to read the research as it comes out, and continually assess which other new LLMs are worth calling out to, and for which purposes. Eventually the API calls will become bi-directional requests to exchange knowledge and insights, i.e. multiple models talking to each other, continually learning.
Thank you for your excellent work. Can you let us know when the 70B weights will be released? I'm really looking forward to trying them out with my own coding project.
I used to use Phind for couple of months. I liked the UI improvements but the slow limited free GPT4 and fast lackluster Phind model turned me off. I tried Bing and it wasn’t worse, had more free searches per day.
They can be, but like with everything copyright-related for copyright to apply there need to be “creative work” involved. Which, for something that has been translated countless times in all possible directions, is going to be much harder than for a first translation.
I have a question because I do not understand how the models work: Are they able to create code themselves, or does code ALWAYS come from a specific source?
I assume that if I ask for a complex sequence in RXJS operators, that comes from the model inferring the code from lots of examples and docs. But if I ask for something really specific that might just come from a stackoverflow article or GitHub repo. The ambiguity about the sourcing is the main thing that makes me itchy about “AI”.
What you'll see in tools that have any exposure to enterprise requirements is an option to say "don't regurgitate your training data". Basically if it generates something that's too similar to any of its input documents, it's thrown away before you see it.
In Github Copilot the option is labeled "Suggestions matching public code". They can offer to block them because they control both the input dataset and the model at inference time. If you download an open source model I don't think you can do it out of the box, you'd need to have that input dataset to be able to do the filtering.
Occasionally I find GPT4 will blur a response indicating it's reproduced from a specific source and will ask me to rephrase my request/question.
So at least OpenAI has some safeguard in place to not do that. Have no clue how that behavior is determined or whether or not other providers do similar.
GPT-4 Turbo sucks for coding, it's not even close to GPT-4. [0]
I tried Phind yesterday and it confidently gave me patently false answers, even after prompting about the specific problems. GPT-4 got it right the first time.
For my own daily use, Phind-70B isn't usable. I get massive daily value from ChatGPT/GPT-4.
Thank you for the reply, I'd like to congratulate you on the release, first. I'm a bit of a minimalist with regard to signups, unfortunately, so unless this is a known limit then I'd likely just spectate the thread and be happy for you from a distance.
It seems extremely less lazy than GPT-4, it spits out code until it's done! Liking it a lot so far. Seems to be the only LLM that defaults to creating Chrome Extensions with manifest V3 while every single other LLM defaults to V2 or V1 unless explicitly told so.
Wow, I'm impressed. I pay for GPT-4 and Gemini Ultra, just to try to keep tabs on where the latest and greatest are.
I recently had a slack conversation with some friends, and someone introduced the made up acronym DILCOLTK, in the context of someone talking about being a DINK and mentioning how cheap things were where they lived. A clever human could infer it to be "Dual Income Low Cost of Living Two Kids", but out of curiosity I tried pasting a bit of the conversation into GPT-4 and Gemini Ultra and Groq, and asking what DILCOLTK referred to. I realize by the way these models tokenize the inputs, it might not be quite a fair question because they maybe can't "see" every letter.
GPT-4 gave "Dual Income Low Cost of Living LTK", Gemini Ultra gave "Dual Income Low Cost of Living One Tiny Kid" (lol), and Groq suggested "Dual Income Low Cost of Living One Kid Two Kid", so all were admirably close but none quite right.
But phind-70B just now got it right! Color me surprised and impressed.
I also asked it a SwiftUI question I'd struggled with, and which I asked the other models about, and it did I'd say a bit better there as well.
So I guess I'll have to add this to my list of models to try and keep tabs on!
GPT-4 is supposed to be 8*220B = 1.7T parameters, so it seems unexpected that a 70B model can beat or match it unless it's somehow a much better algorithm or has much better data.
This is much better than expected. Switching to chat is also making it feel better for me. I will compare it to GPT-4 in coding tasks over the next month and may switch after that.
The one thing we've learnt from the past few months of LLM optimization is that model size is no longer the most important thing in determining LLM quality.
A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there.
I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
It's more a comment on the capabilities of smaller models, the quality of output outside of benchmarks is always subjective and you'd need something like Chatbot Arena (https://chat.lmsys.org/) to evaluate it more quantitatively. Even after filtering out the common cheat techniques like merges, there are still 7B and 13B near the top, but yes it's still possible to train models on the evaluation datasets without decontamination.
If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
It kinda is, if you want not just performance on synthetic benchmarks but a good coverage of the long tail. This is where GPT4 excels, and also why I pay for it. Transformers are basically fancy associative memories. A smaller model, much like a smaller search index, will not be able to contain as much nuanced information for some hard, immutable, information theoretic reasons.
Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
I've found that GPT4 (via GitHub Copilot) and Gemini models are better at code tasks like reviewing for logical and functional errors, reasoning about structure and test/edge cases, and refactoring. Gemini is capable of devouring some very large files I've thrown at it.
Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.
Phind-70B is a specialist model, unlike GPT-4. It optimizes for a different function than GPT-4 and therefore needs fewer parameters to learn it.
It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.
I mean, it could be as good or better at a lot of reasoning related tasks and just have less baked in general knowledge, in which case it'd make an amazing RAG model if the context length is reasonable.
Came here to say this: I try to stay away from Google's products and have been using phind and perplexity for the last couple of months. I have to say I am impressed with what you guys are doing and keep up the good work
I think we need a lot better benchmarks in order to capture the real complexity of typical day to day development.
I gave it my typical CI bootstrapping task:
> Generate gitlab ci yaml file for a hybrid front-end/backend project. Fronted is under /frontend and is a node project, packaged with yarn, built with vite to the /backend/public folder. The backend is a python flask server built with poetry. The deployable artifact should be uploaded to a private pypi registry on pypi.example.com. Use best practices recommended by tool usage.
and it generated scripts with docker run commands [1]:
Which, feels more like "connect the dots" or a very rough sketch, that might end up completely replaced. Commands in general seem ok (yarn install && yarn build, poetry build && poetry publish), but the docker run could be better expressed simply as a "image: " attribute of each job. I asked about that and I've been given general "why docker is useful" non-answer.
It also introduced a parallel build stage: frontend and backend are built at the same time, but in my question, I deliberately introduced serial dependency: the frontend code goes into the backend project. The parallel approach would be of course better, if it would correctly construct the end artifact before uploading, but it doesn't do so. Also a bit surprisingly, node install and poetry install could actually run in parallel as-is, but the generated code runs serially.
It uses outdated versions of tools. Python 3.8 seems still ok and used in many online examples due to some compatibility quirks with compiled libraries, but node 14 is more than 3 years old now. Current node LTS is 20.
- Uses python as base image, but adds the node to it (not a big fan of installing tools during build, but at least took care of that set-up)
- Took care of passing the artefacts built by the frontend; explicitly navigates to correct directories (cd frontend ; ... ; cd ../backend)
- --no-dev flag given to `poetry install` is a great touch
- Added "artifacts: " for good troubleshooting experience
- Gave "only: main" qualifier for the job, so at least considered a branching strategy
- Disabled virtualenv creation in poetry. I'm not a fan, but makes sense on CI
I would typically also add more complexity to that file (for example using commitizen for releases) and I only feel confident that gpt4 won't fall apart completely.
EDIT: Yes, gpt4 did ok-ish with releases. When I pointed out some flaws it responded with:
You're correct on both counts, and I appreciate your attention to detail.
> We love the open-source community and will be releasing the weights for the latest Phind-34B model in the coming weeks. We intend to release the weights for Phind-70B in time as well.
GPT4 gave me a regex found on https://stackoverflow.com/a/2787979 (without "), explained it to me and then it successfully added all the necessary unit tests and they passed - I commited all of that to the repo and moved on.
I couldn't get 70B to answer this question even with multiple nudges.
Every time I try something non GPT-4 I always go back - it's feels like a waste of time otherwise. A bit sad that LLMs follow the typical winner-takes-it-all tech curve. However if you could ask the smartest guy in the room your question every time, why wouldn't you?
---
Edit: USE CODE MODE and it'll actually solve it.