In the positives of Phind:

* Phind was able, even eager, to recommend specific libraries relevant to the implementation. The recommendations matched my own research. GPT-4 takes some coaxing to get it to recommend libraries. Phind also provided sample code using the libraries it recommended.

* Phind provides copious relevant sources including github, stackoverflow and others. This is a major advantage, especially if you use these AI assistants as a jumping off ground for further research.

* Phind provides recommendations for follow on questions that were very good. One suggestion to the Phind team: don't remove the alternate follow on questions once I select one. A couple of times it recommended a few really good follow up questions but as soon as I selected one the others disappear.

In the positives of GPT-4:

* GPT-4 gave better answers. This is my subjective opinion (obviously) but if I was interviewing two candidates for a job position and using my question as the basis for a systems-design interview then GPT-4 was just overall better. In many cases it added context beyond my question, recommending things like logging and metrics for example. It seemed to intuit the "question behind the question" in a much better way than the literal interpretation of Phind. This is probably highly case-dependent, sometimes I just want an answer to my explicit question. But GPT-4 seemed to understand the broader context of the question and replied with that in mind leading to an overall more relevant response.

* GPT-4 handled follow-up questions better. This is similar to the previous point - but GPT-4 gave me the impression of narrowing down the scope of the discussion based on the context of my follow-up question. It seemed to "understand" the direction of the conversation in a way that felt like it was following context.

NOTE: this was not a test on coding capability (e.g. implementing algorithms) but on using these AI coding assistants as sounding boards for high-level design and architecture decisions.

Did you find them to be correct?

I've had one glaring error, I can't quite remember the details, but it switched the names/characteristics of two different processes (ie was exactly opposite in what it said); it was something to do with instruction caching and TLB, IIRC. I assumed you'd was a problem with the input corpus not allowing antonyms to be disambiguated.

Anyway, for me it's the best of the LLM tools I have access to and had mostly replaced search engine (Google, Dukgo) for my tech-related work.

I've only used chat.openai.com (free), bing chat, HuggingChat.

I also spent about 20 minutes on this which is why I mentioned this is a first impression. I'll leave it to researchers to develop a "relevancy" metric and objectively apply it.

In my experience, the sources were sufficiently relevant based on its responses. They were about as relevant as equivalent Google queries. Some tiny, tiny niggles, like I was explicit I wanted it to recommend approaches in Go and for one reference I recall related to distributed locking mechanisms it provided a reference to an implementation in Java. However, that is completely fine for me since the context was more about the locking on the database side and not really the implementation in a specific language.

I didn't click every single link (as I mentioned, the citations are copious) but the few I did follow went to relevant articles. I just went back and randomly clicked several more and they all went to pages that exist and mostly relate to the content of the answer. The inline citations seem a bit more on-topic compared to the side bar which does seem more like the links were lifted directly from a search engine.

To be fair there are some lower-quality blog-spammy kinda stuff - more or less the same kind of thing you would get out of Google. But compared to GPT-4, which provides no sources whatsoever, it is an advantage IMO.

It gets to writing the expected code but it still wants to include formatted headings instead of commenting those out so the entire response is executable Python.

As a follow up I provided an example heading with the hash out front. It didn’t work.

Any ideas on how to get it to this? Fwiw, gpt4 often if ignores this request, but only about half the time. When it does it is typically a single block of explanatory text.

For that, I include prose detection and commenting as part of my post processing.

Also, I don’t see it easily, but do you have an API for this or is it intended to be run by the user?

```keyword // code ```

and then I just ignore anything outside of those blocks.

https://chat.openai.com/share/867ff0c4-d4cf-4af9-a785-31a599...

https://www.phind.com/search?cache=ej8pn1dfjjwfr1tgc6ybwhlg

NOTE: there are a few more question/answer blocks in the phind conversation since I was testing out the follow up question feature.

Edit: they are already posted as comment in this thread.

"Give me five papers with code demonstrating the state of the art of machine learning which uses geospatial data (e.g. GeoJSON) as both input and output."

There is no such state of the art. My hand-wavey understanding is that GIS data is non-continuous, which makes it useless for transformers, and also contextual, which makes it useless for anything else. Will defer to actual ML people for better explanations.

Point is, LLMs invariably give five papers with code that don't actually exist - it's a guaranteed hallucination.

Phind was able to give me five links that do in fact exist, as well as contextual information as to why these five links were not papers with code doing ML with GIS data. This is by far the best answer to this question from an LLM I've received yet.

The code model isn't trained to retrieve papers/articles, it's meant to complete code. Whether or not you find hallucination in a unrelated task isn't particularly interesting.

ChatGPT 4 without web browsing: https://chat.openai.com/share/7e11b4a6-52f2-441a-8614-7266c3...

> There is no such state of the art

Some GIS work uses vector data: points/lines/polygons representing features (e.g., the location of roads or the outlines of buildings), which can be stored in formats like GeoJSON or WKT. But other work uses remote sensing data/satellite imagery that can be stored in raster formats like GeoTIFF - essentially TIFF image files with additional information stored to georeference them.

You can totally do machine learning on satellite imagery where both the input and output are geospatial data (e.g. to categorise land use - the inputs are multispectral images and the outputs can be images where the value of each pixel represents the identified land use).

You can also use machine learning for tasks like building footprint detection/delineation (e.g., [1]) based on satellite imagery. The output from such a pipeline can be a set of polygons, which could be saved as GeoJSON.

I'd consider either of theses to be examples of "machine learning which uses geospatial data (e.g. GeoJSON) as both input and output".

[1]: https://azure.microsoft.com/en-us/blog/how-to-extract-buildi...

Could you please provide me with a query for my PostgreSQL table? It consists of three columns: 'id,' 'teaser,' and 'full_text,' all of which are of type 'text.' Most 'full_text' entries are articles around 1000 words in length. I'd like to update the 'teaser' column for each row with the first approximately 200 words from the 'full_text' column. How can I achieve this?

Phind:

  CREATE OR REPLACE FUNCTION get_first_n_words(text, integer)
  RETURNS text AS $$
  DECLARE
     word_count integer := 0;
     word_start integer := 1;
     word_end integer;
  BEGIN
     WHILE word_count < \$2 AND word_start <= length(\$1) LOOP
         word_end := position(' ' in \$1 from word_start) + 1;
         IF word_end = 0 THEN
             word_end := length(\$1) + 1;
         END IF;
         word_count := word_count + 1;
         word_start := word_end;
     END LOOP;
     RETURN substring(\$1 from 1 for word_start - 1);
  END;
  $$ LANGUAGE plpgsql;

  UPDATE your_table_name
  SET teaser = (
      SELECT STRING_AGG(word, ' ') 
      FROM (
          SELECT SPLIT_PART(full_text, ' ', i) AS word
          FROM generate_series(1, 200) AS i
      ) AS words
      WHERE word <> ''
  )
  WHERE full_text IS NOT NULL;

Also, I ran a few different tests, and every GPT-4 response was superior, but I didn't want to clutter my comment with queries and code.

https://twitter.com/paulg/status/1719657855240815026

No, I do not take these benchmarks seriously and for good reason. They're benchmarks. The only thing that matters is the user's direct experience of the product. And Phind isn't there.

by tweeter trolls?..

If we're comparing benchmark deltas between different fine-tuned variants that share the same base models, that seems like the bare minimum we should expect to come along with performance claims.

Even I am to grant that their subscription product has some secret sauce they want to keep close to the chest (ignoring for a moment their paid product is GPT-4 based), not doing the same for all the models they release to the open source community free of charge with a commercially-permissible license seems suspect.

I realize this sort of open source contribution is mostly for marketing purposes, but being critical of the performance claims I think is still valid nonetheless.

UPDATE your_table SET teaser = substring(full_text from '(\S+\s*){1,200}')

Is that with batching? If so, thats quite impressive.

> certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4.

Some of this is sampler tuning. Y'all should look at grammar based sampling (https://github.com/ggerganov/llama.cpp/pull/1773) if you aren't using it already, as well as some of the "dynamic" sampling like mirostat and dynatemp: https://github.com/LostRuins/koboldcpp/pull/464

I think these should work with nvidia's implementation if you just swap the sampling out with the HF version.

BTW, all this is a great advantage of pulling away from OpenAI. You can dig in and implement experimental features that you just can't necessarily do through their API.

Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?

100 tokens/s on the user's end, on a host that is batching requests, is very impressive.

With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request N, which can help a lot at scale.

I suppose one could get decent utilization by prompt processing one user while generating tokens for another.

ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100:

https://github.com/turboderp/exllamav2#performance

I didn't test codellama, but the 3090 TI figures for other sizes are in the ballpark of my generation speed on a 3090.

100 tokens/s batched throughput (for each individual user) is much harder.

That is quite an accomplishment, I am impressed

The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE (e.g. position interpolation) are post-hoc tweaks (with optional tuning) to an entirely vanilla RoPE implementation.

Additionally Apple researchers seem to be playing with "Attention Free" variants.

https://news.ycombinator.com/item?id=38089888

https://news.ycombinator.com/item?id=38090442

If you want to show the warts of all these LLMs, ask it about ZFS if you know enough to spot the commonly parroted misinformation that plagues the internet.

IMHO, these systems look super useful if they're citing sources and they're worthless without.

https://www.phind.com/agent?cache=cloeowfla000dl1084ermly3c vs https://chat.openai.com/share/4147da33-3669-4657-88fa-3a9dfc...

Might not be representative of the whole thing, but it went on about random things I didn't ask about, and just basic information I already knew

Using the Phind Model in the default search seems to work well: https://www.phind.com/search?cache=ln6dpdtv5auwn4cq1ofg3gs9

Way way better, I'm stunned. Congratulations on this

You see this with Bing search on ChatGPT as well, and I’ve seen it in my own projects.

> Llama 1 supports up to 2048 (2K) tokens, Llama 2 up to 4096 (4K), CodeLlama up to 16384 (16K). [0]

This is wild to me.

The token window is one of the limiting factors for having an AI that can actually remember you and past conversations. Having a large window is key for future AI applications that involve long running conversations (weeks, months, years). The tech is already very impressive, but imagine it as it becomes more like an actual pair programmer and remembers all the various things it's learned and worked on with you in the past.

[0] https://huggingface.co/docs/transformers/main/model_doc/llam...

On cast off/cheap workstation/server hardware.

> Text Embeddings Reveal (Almost) As Much As Text. ... We find that although a naïve model conditioned on the embedding performs poorly, a multi step method that iteratively corrects and re embeds text is able to recover 92% of 32-token text inputs exactly. We train our model to decode text embeddings from two state of the art embedding models, and also show that our model can recover important personal information (full names) from a dataset of clinical notes.

https://arxiv.org/abs/2310.06816

Also, I actually have several top NLP conference publications, so I'm not some charlatan when I say these things. I've actually physically used and seen these techniques improve LLM recall. It really actually works.

Here's more examples of low hanging fruit. The proof in that they work is in the implementations which I provide. You can run them, they work!: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

Check yourself before you try to check others.

They do not. Sentence transformers aren't new, and have well-known trade offs. What source or line of reasoning misled you to believe otherwise?

> Here's more examples of low hanging fruit. The proof in that they work is in the implementations which I provide. You can run them, they work!: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

This...is your blog about prompt engineering. What do you believe this "proves"? How have you blown away current production encoding or attention mechanisms?

You ain’t seen nuthin’ yet…

For example, the researchers working hard on better text sampling techniques (i.e. https://arxiv.org/abs/2202.00666), or on better constraint techniques (i.e. like this https://arxiv.org/abs/2306.03081), or on actual negative prompting/CFG in LLMs (i.e. like this https://github.com/huggingface/transformers/issues/24536) are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today. They are all laboring in relative obscurity.

HN, and the NLP community have some serious blindspots with knowing how to exploit their own technology. At least someone at Andreessen Horowitz got a clue and gave some funding to Oogabooga - still waiting for Automatic1111 to get any funding.

Something interesting that I heard from people trying to memorize things better is that memory “storage space” limits for people are essentially irrelevant. We’re limited by our learning and forgetting speeds. There’s no evidence of brains getting “full”.

Think of it like a giant warehouse of plants, with one employee. He can accept shipments (learning). He can take care of plants (remembering). Too long without care and they die (forgetting). The warehouse is big enough that it is not a limiting factor in how many plants he can keep alive. If it was 10x bigger it wouldn’t make a bit of difference.

It's just that, unlike these AIs, we're capable of online learning.

And if you want vim/emacs to keep living, then you should spend time helping! Create your own extensions, maintain/contribute to existing ones, etc. They will only die out when the last person actively contributing to them stops, so keep the chain of people going :)

While I agree, at the very least IntelliJ stood up on its own as a good IDE. I cut my baby teeth on Eclipse, and as soon as I realised how good IntelliJ is, I jumped ship without looking back. The same can barely be said about VS Code.

There's an argument that music and the arts are dumbed down by the fact that, for instance, making an album worth $10 to millions of people pays way better than making an album worth a million dollars to tens of people, since the album is going to get priced at $10 one way or the other. It only just now occurred to me that the same phenomenon applies to tools.

:'<,'>y|call system('firefox <url>?q='.shellescape(@*).' &')

The only problem left is that the text is not urlencoded.

There probably is some elegant way to urlencode it. But I did not come up with one yet.

Working with phind needs to be available in pycharm for me considering switching from gpt4 to phind. Chatting with phind on my local files is the feature I am looking for.

[1]: https://github.com/s-kostyaev/ellama

[2]: https://github.com/ahyatt/llm

https://github.com/huggingface/llm.nvim

- Dark mode is hard to read, the answer text font has too much weight and brightness which makes it hard to read long paragraphs of non-code text. Light mode is obviously too bright overall, but it's already nighttime where I'm at so maybe tomorrow at noon I'll have another opinion. I'd preferred gray (dark, ie OpenAI) and sepia (light, ie HN) as backgrounds when long lines of text are involved.

- Pricing page and ties to GPT-4: what does "500+ best model uses per day (GPT-4)" mean? What's the "GPT-4" part for? I saw I can pick GPT4 as a model on the landing page, but I just don't get the best model/GPT-4 thing. Is Phind announcing it's a competitor but also proxies GPT-4? Sorry, I'm not up-to-date on GPT-4 "resellers" and the story behind Phind, it's just weird when it announces it "beats GPT-4" then the pricing is about GPT-4 usage.

I always shy away from stuff like this because I view it as one of two things. Either I'm getting ripped off if I pay for Plus, because 8x the cost to me means your margin is huge, or I'm getting subsidized by you with the Pro version which means I can't rely on it lasting long term.

I also dislike daily limits for search. My search usage isn't uniform day-to-day. I might go most of the month without searching for anything and then do a ton of searching over 2-3 days when I'm trying to learn something. So I'll be idle most of the month and then not have enough searches on the days I actually want to use it.

I prefer the model used by a lot of pre-paid services. Let me deposit a chunk of money (ex: $20-50 minimum) and charge me per search until my money is gone. That way I'm not "losing out" if I don't use it every day and I can "burst" as high as I want when I'm trying to learn something.

If the pricing is based on a certain amount of loss (on my side) from the use-it-or-lose it model, I don't like that. I want simple, fair pricing, not a complex pricing scheme where the primary purpose is to get me to overpay for my usage.

We understand that the incentives of setting daily limits for search aren't great, which is why the Phind model is unlimited for free. GPT-4, however, is unfortunately too expensive for us not to charge past a certain usage threshold.

https://www.phind.com/search?cache=wgyz13tg4jkbl9pklptmpds5

edit: looking at it again, I think the $15/mo is actually just for people who wants Phind "private", so that their data is not used for training.

ChatGPT 4 does not take 50 seconds to answer, so I don't understand this comparison.

I've asked it how to scp a file on Windows 11 and it'll take a minute to tell me all the options possible.

If this takes 1/5th the time for equivalent questions, I'd consider switching

If so, consider adding one of the “just get to the point” prompts. GPT4’s defaults have been geared towards public acceptance through long-windedness which is imo entirely unnecessary when using it to do functional things like scp a file.

Just trying to provide helpful feedback for you, this would have been a great comment, except for the "LOL" at the beginning that was unnecesary and demeaning.

By keeping the prompt short, it starts generating output quicker too.

Also, use case thing. It is very likely the case that for certain coding use cases, Phind will always be faster because it's not designed to be general purpose.

ChatGPT4: 14 seconds

phind with "pair programmer" checked: 65 seconds

phind default: 16 seconds

It lets you specify verbosity from 1 to 5 (e.g. "V=1" in the prompt). Sometimes the model will just ignore that, but it actually does work most of the time. I use a verbosity of 1 or 2 when I just want a quick answer.

https://imgur.com/a/iqxOJUV was 6.5 seconds.

https://imgur.com/a/pQFfWli was 15.

You can tell they're GPT-4 because the logo is purple (the logo is green when using 3.5).

Which yes, is missing a lot of detail (you could/I have feed/fed in a datasheet).

But Phind goes off on using pyserial (?!), and GPT4 assumes amaranth is a hypothetical CCD control library and makes a useless class control CCD using the hypothetical library.

Edit - Phind at least acknowledged that amaranth exists, unlike GPT4 with this prompt: "Write amaranth code that can be used to control the readout of a frame from a kodak CCD using an lattice FPGA with 4096 columns and 2048 rows. Assume the design will be hooked up to a larger litex SoC "

Controlling a CCD is actually pretty easy, I built (very simple, but working) controllers for several CCD chips in undergrad doing research for the ATLAS detector. You just clock a rows out basically, N columns times. Reset first. I'd expect an senior undergrad EE student to be able to design a simple core in a few class projects.

https://pastebin.com/BCAthV8y

Especially for agents that do function calling.

And Cursor is just the start - there will be innovative workflows built on top of APIs you can't predict. You're missing out not having developers build an ecosystem for you.

I agree with the assessment about consistency being its major flaw. While with GPT4 I can continue a conversation for quite long, Phind easily looses the required context. Perhaps it has to do with summarization capabilities, or messing with the context window has these types of side effects.

They're smashing it and can't do enough if you report an issue, also they have started a weekly voice call to discuss algos and such with senior devs, like a surgery, only 10 people join at moment

Don't think I've ever recommended anything as much as I have these guys in last couple of months

Note that I'm not really a power user of these GPT style tools- here are my questions:

Is it possible to get right to the code without the ELI5 and general information?

Do you guys offer an API? I was browsing on my small iphone so maybe I missed this info.

Could you give an overview for someone like me how something like phind works technically? You mentioned those H100s, but at a very high level without revealing any "secret sauce" how does this GPT work from my input to getting a response?

Good luck!

In any case I am happy there is some competition and that it has come from a more pragmatic scrappy space than one of the multiple billion dollar funded places.

When fb released Llama they obviously gained a huge amount of developer goodwill but it also required them to invest a serious amount of their own developer time to engage with the community.

I'm asking the community what it can offer the company? Or is this just self-abnegation by the company that releases the model?

https://huggingface.co/Phind/Phind-CodeLlama-34B-v2

GPT-4 on the other hand provided a good answer for both questions. Also I guess the UI is buggy w.r.t code formatting, it things the following line is a code and switches to a code block.

``` You are given an array prices where prices[i] is the price of a given stock on the ith day. ```

The only downside for GPT-4 for me right now, is its slowness.

Only exception I found is with some of the Leetcode Premium questions, you might have to actually type in the problem statement, but it's still very likely that multiple solutions have been ingested from GitHub and elsewhere.

``` You are given an array prices where prices[i] is the price of a given stock on the ith day.

Find the maximum profit you can achieve. You may complete at most two transactions.

Note: You may not engage in multiple transactions simultaneously (i.e., you must sell the stock before you buy again).

Write Python code to solve this: def maxProfit(self, prices: List[int]) -> int: ```

Output:

``` It seems like you want to find the maximum profit that can be achieved by buying and selling stocks, with the constraint that you can only make at most two transactions. Is that correct?

Could you please provide some example input and output to help me better understand your requirements? ```

I also tried a more basic prompt, but the output is not what I'd consider good code.

Can you maybe share some examples where we can see how it exceeds GPT-4's capabilities? Thanks!

I want to make an interactive plot in Colab where I can show

X axis is interest rate of a 15 year mortgage. Y axis is the relative advantage of buying a house vs. renting in terms of total net worth at 15 years.

Assume a monthly budget for renting + investing or buying a house of 10k

Plot different lines for a few different market returns.

Make a slider that controls the total size of the loan.

Chat gpt actually implements it

I've been searching for an obscure connector -- the 8-pin connector you'd find on the cable that delivers power to a GPU, but in a form that can be wave-soldered. I've spent hours searching all the big electronics distributors -- no luck. This thing found it in seconds.

https://www.phind.com/search?cache=a7e9u5l5aw1r8ufls0icpb63

This is a very common connector but in a highly unusual form-factor. Molex refuses to make wave-solderable versions of it.

Edit: the first link does not lead directly to the obscure connector, but to the website of a company that does sell it. Here is the obscure connector: https://www.moddiy.com/products/Special-Mini-Low-Profile-ATX... Maybe it just got lucky.

On the other hand, it hallucinated the crap out of a very straightforward question "how do i connect the wake# pins when bifurcating a pcie port?" -- the answer is that it's an open-drain pin so (unlike the clock pins which need a buffer chip) you just wire them both together:

https://www.phind.com/search?cache=zf9witr85q740l4s3vjwzf01

Then it tried to write a bunch of code for an obviously-not-coding question. Not so great.

https://www.moddiy.com/products/Special-Mini-Low-Profile-ATX...

I'm just elated I finally found out where to buy these damn things from.

will there be api access soon?

also: will it be open-source at some point? thanks

API access is on the roadmap but we have no time estimates for when we will build it. We're trying to not get distracted from our main mission :)

So Phind's main mission is to overtake Google right? ;)

Claude-2, correct response and rounds the percentage - makes a nice assumption for me.:

    SELECT state, 
           count(*) AS people_count,  
           round(100.0 * count(*) / (SELECT count(*) FROM onboardings), 2) AS percent
    FROM onboardings 
    GROUP BY state
    ORDER BY people_count DESC;

    WITH state_counts AS (
     SELECT state, COUNT(*) as count
     FROM onboardings
     GROUP BY state
    ),
    total_counts AS (
     SELECT COUNT(*) as total
     FROM onboardings
    )
    SELECT sc.state, sc.count, (sc.count::decimal / tc.total::decimal) * 100 as percentage
    FROM state_counts sc, total_counts tc
    ORDER BY sc.count DESC;

I gave this as an example

“create a CDK typescript app that deploys a lambda + API Gateway where the lambda works with Get request and a dynamodb table. The lambda should have permission to read and write to the Table”

It wrote the code perfectly. I wanted to see if it was trained on the AWS APIs.

Maybe I'm not knowledgeable enough. But asking questions I already know the answer of has no real life use case other than testing the model. Which of course might be a valid use case for some.

Having a way to let the user specify his own level of knowledge might help to receive answers that are better tailored to the user asking the question.

My dad always used to say: Everything that can be said, can be said simply. I prefer top-down structured, short and thoughtful responses.

> I need a typescript function which takes in an object with an id string property and a string message property, and also takes an array of search strings, and returns a mapping of search strings to matching message ids

The response I got was close, but it assumed that each search string would match only one message, so it returned Record<string, string>. I fed this to GPT-3.5 and it answered 10x faster with the correct return type.

This is a slightly tricky example, because it requires the model to infer that multiple message matches are possible. But I think that it’s interesting that ChatGPT nailed it despite not using any chain of thought.

Your prompt is wrong. You want a function that takes an array of id/message objects, not an object.

It's quite impressive that GPT is just able to correct for that. As a human, I would first ask what you actually mean, because your prompt appears to be unclear.

When I ask it as one statement it performed ok, but if I made more specifications with follow-up statements, it kept trying to go down one path even though I told it to do it a different way. A solid start but it definitely needs some improvements, IMO.

E.g.: the efficient and robust file change monitoring on Windows is to read the NTFS change journal. For a single process lifetime there are other change notification APIs as well. Most software does neither and is either very slow or misses changes…

And I also can't wait to see how much Phind will improve further if the Glaive dataset is added onto it.

Edit: Contrastive search, dynamic temperatures.

(An even more interesting question would be to compare ChatGPT GPT-4 with Phind GPT-4, i.e. GPT-4 with relevant web results in context.)

[1] https://www.phind.com/search?cache=u3mnj3iwmjvgqlyf60bnbqo1

I asked 5 different, and increasing explicit, variations of the following question: "Can you generate HTML and CSS for a JPG mockup I'm going to give you?"

Each time it answered along the following lines: "Sure, here is how you can create HTML and CSS from a JPG mockup. Follow this process..."

In my experience this never happens with GPT-4.

However, my point was that Phind's answer was worse than a No, or a hallucinated attempt would've been. By saying "Yes, here is how YOU can do it...", it left the impression that it didn't even understand the question.

Does it? I don't see any evidence of this strong claim in your post, and I think it's quite deceptive how the only link is to a benchmark of open source models (which doesn't include GPT-4). I've tried Phind a few times in the past when it made equally strong claims and been somewhat unimpressed. (To be fair, comparing anything to GPT-4 is tough!) I think it would strengthen your position significantly to simply say that you're the best of all open-source models.

To be honest though I've been completely ruined by https://cursor.sh/; copying and pasting results back and forth from a web UI to my IDE is so painfully slow when you do it tens or hundreds of times that I don't think I would be able to go back. I'd be happy to try out a Phind extension that has similar UI/UX if you ever make one.

ChatGPT4 gave me a solid answer hitting all the points I wanted. Phind din't get the account handling correct, didn't address regions, and didn't handle pagination.

"Write a python based script that uses boto3 to query AWS Route53. It should print a list of every record for a given hosted zone ID."

ChatGPT4 did exactly as requested with pagination, and even smartly decided to use "input" so I could give it a zone ID at run time. Phind didn't handle pagination, or do ANY error handling. It was also slower than ChatGPT4 to generate currently, and it wasn't in a single block of copy/pasteable code.

ChatGPT's solution worked without modification. Copy-Paste. Run.

We do have issues with consistency sometimes -- please try regenerating if that is the case.

Consistency issues can be caused by a wide range of factors from inference hyperparameters to prompting.

Basically:

"How can I send network control commands to an AppleTV in C#"

They always make up some nonexistent library or gives an example using some nonexistent API.

“Make me a billionaire… I’m still poor! Bad AI!”

You need to collaborate with the AI, use it to help with each small step of the problem, with input references provided.

To a degree Phind can do the reference chasing for you, but it’s not magic.

Someone is doing it in python here:

https://pyatv.dev/

GPT-4 actually sent me here:

"Here is an example of a C# library that implements the HAP: CSharp.HomeKit (https://github.com/brutella/hkhomekit). You can use this library as a reference or directly use it in your project."

Which, to no surprise based on my experiences with LLMs for programming does not exist and doesn't seem to have ever existed.

I get that they aren't magic, but I guess I am just bad at trying to use LLMs to help in my programming. Apparently all I do are obscure things or something. Or I am just not good enough at prompting. But I feel like that's also a reflection of the weakness of an LLM in that it needs such perfect and specific prompting to get good answers.

Assuming a C# library even exists for what you’re doing (maybe not!) then still the best use of AI is to troubleshoot specific issues given an almost working piece of code as input.

Ask it to explain why something doesn’t work instead of asking it to do your job for you wholesale.

PS: GPT 4 (you are using the best coding AI, right? Right?) can get you going quickly:

“There are several libraries available for controlling Apple HomeKit from C#. One such library is *HapSharp* ². It is a .NET implementation of the HomeKit Accessory Server that allows you to create your own custom HomeKit accessory on a Raspberry Pi, Mac computer, or any other platform that can run Mono ².

Another option is *HomeKit* ¹. It is a native C# library for Apple's HomeKit Accessory Protocol. However, it is not a complete implementation and does not work ¹.

I hope this helps!

Source: Conversation with Bing, 31/10/2023 (1) netonjm/HapSharp: HomeKit Accessory Server .Net bridge! - GitHub. https://github.com/netonjm/HapSharp. (2) GitHub - ppumkin/HomeKit: Native C# Libary for Apple's HomeKit .... https://github.com/ppumkin/HomeKit. (3) homekit-accessory-protocol · GitHub Topics · GitHub. https://github.com/topics/homekit-accessory-protocol?o=asc&s...