There have been a spree of recent experiments with LLMs solving logic puzzles (specifically Cheryl's Birthday). I wanted to replicate and repeat the tests from [0] with more LLMs. For reference, that article tested whether the trained models handled obfuscation of the text so that verbatim discussions of the solution were less likely to appear in the training corpus.
Then I wanted to move further and test whether LLMs were prone to distraction with extraneous and irrelevant data. In a world where RAG may pull in "compromised" data, I wanted to see if LLMs could ignore cruft or if it would alter their answer. TL;DR - it altered the answers.
o1 dropped as I was making graphs etc so, I included the results from testing it as an additional section. It was still distractable but was more capable in the obfuscated case.
Forgive the bait headline, I'm still trying to find the best balance of information and marketing for posts like this. Suggestions welcome on that front.
Related to this in asked LLMs to directly solve the same riddle but then obfuscated the riddle so it wouldn’t match training data and as a final test added extraneous information to distract them.
Outside of o1, simple obfuscation was enough to throw off most of the group.
The distracting information also had a relevant effect. I don’t think LLMs are properly fine tuned for prompters lying to them. With RAG putting “untrusted prose” into the prompt that’s a big issue.
How do you know whether the discrete events are a more fundamental representation vs a higher level representation that your training and discipline produces?
It’s a great question. I have come up with two answers (though I am far from an expert):
1) this is empirically verifiable; just do an RCT where you teach people a meditation technique for attention without prompting, and see what they observe. (I have heard comments from aspiring meditators like “I tried meditating but after a while I could not find “the breath” because it broke apart into a stream of individual sensations”) - but I do worry that techniques like “noting” smuggle in an atomizing assumption, whereas other techniques like whole-body perception or Metta might lead you to a more unifying viewpoint if practiced exclusively.
2) maybe it doesn’t matter if it’s “more fundamental”; if you wire your brain to deeply believe that it is, then a bunch of positive effects occur, and that’s the goal of the whole exercise. The words “this is more fundamental” are just a cue to help you to shift. This feels less palatable to me but I haven’t seen the rewards, and if they were as good as promised maybe this would be justified.
Anyway, I’m not sure many Buddhists would endorse 2), even among the secular / non-religious/ scientific minority of the community.
You don't. Everything is mental fabrication, including calling some mental fabrications illusion. You train your mind to fabricate things that you prefer. Whether they are more fundamental or higher-level representations doesn't matter.
> Release the data, and if it ends up causing a privacy scandal...
We can't prove that a model like llama will never produce a segment of its training data set verbatim.
Any potential privacy scandal is already in motion.
My cynical assumption is that Meta knows that competitors like OpenAI have PR-bombs in their trained model and therefore would never opensource the weights.
This seems fundamentally different. Filter bubbles show you more of the externally generated content you engage with. These personalizations are trying to predict the content you generate.
While it may serve as a ballast for your personal voice changing over time, the whole point is to learn you not to feed you.
I think it is correct to include practical implementation costs in the selection.
Theoretical efficacy doesn’t guarantee real world efficacy.
I accept that this is self reinforcing but I favor real gains today over potentially larger gains in a potentially achievable future.
I also think we are learning practical lessons on the periphery of any application of AI that will apply if a mold-breaking solution becomes compelling.
> It just means you may have to roll up your sleeves and help create the community you want.
Citation needed. There are so many of these small towns that are hurting surely, you can find a single example or anecdote that backs up the claim that this is a plausible much less obvious solution.
The numbers indicate that levels of poverty are higher in those small towns per-capita in the US [1]. And these studies have yet to include the impact of COVID-19. Anecdotally, every small town I know of saw wages go down and housing prices increase in the pandemic.
Also wages in urban areas are growing faster than rural areas [2]. So your lifetime earning potential is dramatically impacted by being in a smaller town. That may be a good choice for many people but if the argument is that you will be better off financially the numbers don’t support that.
I am early in my journey but I’m stumbling on the basic structure of these models.
Is this structurally a vanilla transformer (or encoder/decoder) with tweaks to the tokenizer, the loss function, the hyper parameters, and the method of training?
Is whatever this is representative of most of the publicized releases? For instance the recent Orca 2 paper didn’t seem to have any “structural” changes. Is there a better term for these distinctions?
I don’t mean to downplay the importance of those changes, I am merely trying to understand in a very broad sense what changes have what impacts.
It's pretty much the same architecture since GPT2, just a bunch of self-attention transformer blocks.
The reason these have been better is because we have more GPU, more data, and have scaled the attention calculations to be linear instead of quadratic, so we can train even bigger models. We've also been finetuning models on higher quality data.
To understand the orca papers you need to understand how models are trained.
Pretraining: this is when we train a model from scratch on all the data that we can get from the internet.
Finetuning: We further train the pretrained model on a specific style. For chat models this is called the instruction finetuning, this is where the model learns to respond in a specific format and align it to be helpful, etc. We do this by giving it a bunch of texts of assistants answering questions and being helpful.
Llama2-chat is a finetune of llama2.
Zephyr-b is a finetune of mistral 7B.
Yi-34B-Chat is a finetune of Yi-34B.
We can also further finetune models by using RLHF and other reinforcement learning techniques.
Most model releases are finetunes of other models, i.e. when meta released the llama models it created a deluge of chat/instruct finetunes from all over the community. The orca papers are essentially finetuning papers, the focus on what kind of data you should feed to models to get the most out of it for following instructions among other things.
My understanding is they are all still transformers. The tweaks are more about quantization that better to generalize over data more efficiently (so less parameters requires) and improvement of the training data/process itself.
Otherwise I'd like to know specifically whats better/improved between models themselves.
If it’s just that the parties don’t trust each other then the cost of HME has to be compared to the current “state of the art” which is contracts and enforcement thereof.
In practice, I don’t think those costs are that high because the rate of incident is low and the average damage is also low.
Yes there are outlier instances of large breaches but these seem like high profile aircraft crashes considering how many entities have sensitive data.
I feel like trust is a spectrum, and the promise of these techniques is that they reduce the need for trust in the first place.
We should consider what kinds of computational tasks today’s responsible parties (or their regulators, or their insurers) think of as too risky to casually trust to third parties under the status quo. For example with my block storage provably unintelligible if you don’t have the HSM I keep safely in my corporate dungeon, I’m comfortable not caring whose racks the encrypted blocks sit on. I’d have to vet those vendors a lot harder if they could read all my super secret diaries or whatever.
And, for that matter, it’s on the service provider side too, right? Even the contractual, spit-and-handshake pinky-swear-based mode of enforcement comes with significant compliance costs for service providers, especially ones operating in regulated industries. Perhaps it’s not too much to hope that effective and efficient HME techniques might reduce those service providers’ compliance costs, and lower the barrier to entry for new competitors.
I’m reminded how even non-tech people in my life became much more willing to trust their credit card details to online retailers once they felt like a little green lock icon made it “safe”. Of course a LOT changed over that same period, but still: the underlying contractual boundaries didn’t substantially change—in the US the customer, then as now, has only ever been responsible for a certain amount of fraud/theft loss—but people’s risk attitudes updated when the security context changed, and it opened up vast new efficiencies and lines of business.
It’s not too much to hope that HME reduces those compliance costs. However, I believe it is too much to assume there will be any material adoption before it can demonstrate that reduction.
Reduction of trust is not a value add, it is a cost reduction. Maybe that cost is blocking a valuable product/service but either that product/service’s value is less than the current cost of trust OR trust has to be far more costly in the context of the new product/service.
It’s only the latter that I find interesting which is why tend to be pretty hard on suggestions that this will do much for anything that currently exists. At best, it will improve profits marginally for those incumbents.
What is something where the price of trust is so catastrophically high in modern society AND HME can reduce that cost by orders of magnitude? Let’s talk about that rather than HME.
Data incidents cause more problems than can easily be resolved with a contract lawsuit. Perhaps the data was siphoned by a 3rd party that hacked your vendor, or a malicious insider at your vendor sold it to a competitor. Sure, you can recoup some losses by suing your vendor for breach of contract, but once the data is leaked, it's never secret again.
And then there's the example of businesses that work with lots of confidential customer data, like banks or doctors. Again, you can sue your vendor for breach of contract if they behave irresponsibly with your data, but your customers may not care; you're going to suffer a hit to your reputation regardless of whether or not the breach was your fault.
You can say it’s insufficient but it is what it costs them today.
I guess the better comparison is that cost in a financial statement plus some expected increase in revenue due to a “better” product.
Again, I think you are correct in your analysis of the improvements but that contributes little to the revenue as explaining the benefit to most customers requires framing your existing product as potentially harmful to them. Educating them will be hard and it may result in an offsetting realization that they were unsafe before and as a result were paying too much.
Not really, you would phrase it to your customers or investors as a way of mitigating risk. You can probably apply a price tag to that risk by estimating the impact of a data incident vs. the likelihood of one happening. Different businesses have different risk appetites, and I would hope that a board or C-Suite is thinking about what level of risk is acceptable for their business.
Mitigating risk is covered in the cost reduction side.
Yes the C-Suite is thinking about and mitigating risk. They probably know the exact number for a given class of risk in terms of current mitigation costs. You have to beat that by a margin wide enough for them to take action.
Even if you know their numbers and know you beat it by enough to warrant the deployment you will still get bumped if someone sells them a path to increasing revenue.
The out I gave was to frame it as value added (more revenue) and that is where you risk devaluing your current product.
If you frame it as cost reduction you are capped in both price and interest by the current, necessarily acceptable, levels of risk and cost of mitigations.
Then I wanted to move further and test whether LLMs were prone to distraction with extraneous and irrelevant data. In a world where RAG may pull in "compromised" data, I wanted to see if LLMs could ignore cruft or if it would alter their answer. TL;DR - it altered the answers.
o1 dropped as I was making graphs etc so, I included the results from testing it as an additional section. It was still distractable but was more capable in the obfuscated case.
Forgive the bait headline, I'm still trying to find the best balance of information and marketing for posts like this. Suggestions welcome on that front.
[0] https://timharford.com/2024/08/ai-has-all-the-answers-even-t...