I appreciate that the author is using very specific meaning for the term “magical thinking”, but even with this, I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking, as (re)defined in the article.
Anyone who spends enough time studying biochem will have an intuitive understanding / acceptance that biology emerges from chemistry, which emerges from physics… But there’s not really any “magical thinking” (as defined in the article) involved.
Emergence of cognitive processes and experiences in the brain is not really any different in this respect. If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.
> Anyone who spends enough time studying biochem...
...knows about how LSD was used to discover DNA. Agreeing it possible is one thing, but to actually expect it? That is magical thinking.
I don't think putting random chemicals in your brain is all that different than putting a bunch of random coin-flips in a training function, so I don't think it's a terrible analogy either: The fact that something amazing comes out of that process is magic.
> I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking
That's common amongst scientists. Words matter, and replacing magic with "emergent" keeps it at arms length, and detachment is important to do science.
But I suggest you play along anyway, instead of pretending they meant "emergent" every time they say "magic" because words matter...
> If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.
I hear you, but it is hard to operate on the assumption you're not special, after all, you're probably more successful than a lot of people, and I bet you think your hard work had something to do with it.
The science says this isn't true, of course, that the colour of your skin, the postcode you were born in, that the wealth and other genetics of your parents, are all greater predictors of success, and you probably know this, but if you're a human being you can't help but think that you have willpower and ethics, even if you don't believe in a soul.
Or another example: if you've ever looked at someone who behaved badly, and said you wouldn't have done that? Of course, if you really did have their life, their parents, their postcode, their skin, their abuse and traumas over the years, their joys and friends, stories of parties and life, and so on, well then you definitely would have, or you've got to believe you're special.
This kind of magical thinking has kept you alive- it's provided impulse to the bag of biology and chemistry to twitch and react its way through your life- something you've got to admit sounds a lot less fun when I use science-words to put it that way.
“Would I have invented PCR if I hadn’t taken LSD? I seriously doubt it […] I could sit on a DNA molecule and watch the polymers go by. I learnt that partly on psychedelic drugs.”
> Is there some sort of allusion to the meritocracy trap or fundamental attribution bias? … I'm just confused.
I have no idea what led you to that idea and I’m very sorry to have confused you, what else can I say?
> Or another example: if you've ever looked at someone who behaved badly, and said you wouldn't have done that?
Is where OP perhaps got fundamental attribution error from.
> ...you're probably more successful than a lot of people, and I bet you think your hard work had something to do with it.
Is very similar to the myth of meritocracy some folks believe in.
---
In sum, you intend to point out that cognitive biases (or perhaps, cognitive dissonance) are also a form of magical thinking? Like OP, I find it hard to reconcile that with LSD/DNA discovery. In my view, biases mostly fool you; but they don't quite emerge out of an insight (magical thinking), as the article describes it.
My thought about emergence is that it is not a phenomenon in its own right, but a category of phenomena that are identified subjectively. A lot has been written describing the wonders of emergence. Synthetic model systems have been developed with seemingly emergent behaviors, without leading to novel, empirically testable predictions. So far the only empirically testable models leading to emergent behaviors such as we observe in nature, are in fact the laws of physics and chemistry, more or less as we know them today.
To me, "magic" means that a phenomenon is inherently inaccessible to naturalistic inquiry, not merely that such inquiry is temporarily stalled due to inattention, lack of the right insight, or insufficient technology.
To that short list I would add some purely mathematical topics such as cellular automata (which show great promise towards helping formalize the concept of emergence IMO), but also biology, ecology, neuroscience, psychology, sociology, economics... There's layers upon layers of emergence all around us.
Even just within "physics" there are probably multiple layers of emergence (a striking and popular example is particle-like behaviour arising from wave-like fields). There is "reasonable speculation" as to whether space, time, gravity might all be emergent rather than fundamental...
Indeed, that's fairly widely accepted. What I'm not sure about is whether the study of emergence has yielded new scientific insights yet.
The author mentions Wolfram, but I think fumbles around for a reason why scientists were ambivalent about Wolfram's work. The reason is that clever and even surprising demonstrations of cellular automata have not yet yielded any connection to empirical science, thus it's premature to call it a "new kind of science." Not because scientists are opposed to computational models. Physics used up its supply of "closed form equation" models roughly a century ago, and has fully embraced computation. We even suffered through a few decades of grinding through computations by hand, waiting for computers to be invented.
If you stretch the definition of cellular automata a bit, you end up with models that divide regions into a grid of cells, with evolution rules that lend themselves to iterative computation. Examples include finite element analysis and climate modeling.
Personally, I think it's sufficiently interesting that "emergence" happens and that we have multiple examples of it. I'm not hung up on whether the study of it will lead to new testable predictions (though that would be wonderful). I think in the worst case, it already gives us a vocabulary to explain that some unintuitive or surprising phenomena of a certain category are normal, and that higher-order patterns will occur naturally even from very simple systems. In the worst case, it's merely a word for an abstract pattern we observe all around us.
I'd wager the vast majority of laypeople cannot conceive how humans (or life) came to be without some sort of divine intervention. I certainly don't think anyone has the answer to that haha... But having this vocabulary, and "mere knowledge" of the existence of various examples of emergence can replace this incredulity with a more grounded appreciation of the inherent beauty and wonder of nature. I think the concept of emergence is profound in itself, because of how it can shift our perception of our own place in the world.
"I've never seen anyone say ChatGPT feels like magic"
This is a curious line - I have, and I personally have that reaction when I use it. It might even be the most magical technology I've ever encountered.
Literally in the next sentence the author also remarks that he finds it odd:
> “That makes me curious.”
He finds it curious because he also finds the technology is magical.
Eventually he concludes:
> “Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it – a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.”
> “When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights – things the reader has never heard before – on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic.”
While I personally don't think ChatGPT feels magical, I know many people who feel this way. It's a really common opinion, especially among non-programmers.
FWIW, apenwarr is one of the most intelligent humans I've ever met. When people bring up 10x programmers I think of him. I don't really have much to add, I just wanted to mention that for context.
> Since we now know what magic means, we know what magical thinking means: a tendency to interpret something as magic instead of trying to understand it.
Mind you, people often use the phrase in a pretty loose way as a way of dismissing an argument, but it does seem to be based on the technical use, and not the meaning Apenwarr puts forth.
> In short, in software engineering, we acknowledge that failures happen and we measure them, characterize them, and compensate for them. We don't aim for perfection.
If software engineering revolves around writing software complex enough to have bugs not understood, or accepting memory safety errors as a natural byproduct of programming (rather than programming mistakes to be exterminated on sight, and made impossible by design unless it prevents writing useful programs), I'd be more effective and happier practicing applied math than engineering.
It's also accepting that performance is variable, that the network is not reliable, that machines can fail at any time, and that memory corruption happens.
When you build systems at scale you need to worry about these things.
Programming language API boundaries don't make real-time performance guarantees, not even in Rust. We hope to get typical performance. In a mathematical context (for example, proving programs correct), you often ignore these things and worry only about whether the program will give you the right answer, eventually, when the hardware works.
Placing emerged complexity/magic in a bucket together & saying science despises them is a novel & revelatory point of view. Avery is such a treasure!
Lovely idea towards the bottom too,
> Or maybe we need to draw a distinction between producing insight and recognizing insight.
I personally identify with this & see it as super core a distinction. I'm reasonably bright & can find some clever ways of doing things even if I haven't any learnings here. But like 90% of what I love & appreciate about self is a resourcefulness & ability to recognize & integrate insight.
Many people keep saying the brain isn’t magic so blah blah sentience etc, but if that’s the case, we cannot claim LLMs are magic either, or any “emergent properties”, else you do believe in magic. Which is it, magic or no magic? We can not have it both ways.
Maybe the idea of magic shouldn’t keep being referenced, we should refrain from comparing and just apply equal respect to all beings and phenomena and just admit some things are, and always will be mysterious and move on with life ?
It’s possible to build a cache without any insight, but to make a cache that works well in a given domain requires knowledge from outside the world of caching itself. The same is true of summarization and compression.
thanks, a good read, I'm always curious to go back and look at early knowledge system innovators who were looking at the 'causes' of 'breakthrough hypothesis' and looking to then 'engineer' a surrounding system to support more frequent breakthroughs.
> Any sufficiently advanced technology is indistinguishable from magic.
I had a lecturer who used to counter "if something ever seems like magic it means you're not asking enough questions."
That stuck with me, although the article goes on to counter it fairly well.