as ever, any task that has any sort of safety or security critical risks should never be left to a “magic black box”.
human input/review/verification/validation is always required. verify the untrusted output of these systems. don’t believe the hype and don’t blindly trust them.
—
i did find the fact that google search’s assistant just parroted the crafted/fake READMEs thing particularly concerning - propagating false confidence/misplaced trust - although it’s not at all surprising given the current state of things.
genuinely feel like “classic search” and “new-fangled LLM queries” need to be split out and separated for low-level/power user vs high-level/casual questions.
at least with classic search i’m usually finding a github repo fairly quickly that i can start reading through, as an example.
at the same time, i could totally see myself scanning through a README and going “yep, sounds like what i need” and making the same mistake (i need other people checking my work too).
> any task that has any sort of safety or security critical risks should never be left to a “magic black box”.
> human input/review/verification/validation is always required.
but, are humans not also a magic black box? We don't know what's going on in other people's heads, and while you can communicate with a human and tell them to do something, they are prone to misunderstanding, not listening, or lying. (which is quite similar to how LLMs behave!)
> at the same time, i could totally see myself scanning through a README and going “yep, sounds like what i need” and making the same mistake (i need other people checking my work too).
yes, us humans have similar issues to the magic black box. i’m not arguing humans are perfect.
this is why we have human code review, tests, staging environments etc. in the release cycle. especially so in safety/security critical contexts. plus warnings from things like register articles/CVEs to keep track of.
like i said. don’t blindly trust the untrusted output (code) of these things — always verify it. like making sure your dependencies aren’t actually crypto miners. we should be doing that normally. but some people still seem to believe the hype about these “magic black box oracles”.
the whole “agentic”/mcp/vibe-coding pattern sounds completely fucking nightmare-ish to me as it reeks of “blindly trust everything LLM throws at you despite what we’ve learned in the last 20 years of software development”.
Sounds like we just need to treat LLMs and humans similarly: accept they are fallible, put review processes in place when it matters if they fail, increase stringency of review as stakes increase.
Vibe coding is all about deciding it doesn’t matter if the implementation is perfect. And that’s true for some things!
> Vibe coding is all about deciding it doesn’t matter if the implementation is perfect. And that’s true for some things!
i was going to say, sure yeah i’m currently building a portfolio/personal website for myself in react/ts, purely for interview showing off etc. probably a good candidate for “vibe coding”, right? here’s the problem - which is explicitly discussed in the article - vibe coding this thing can bring in a bunch of horrible dependencies that do nefarious things.
so i’d be sitting in an interview showing off a few bits and pieces and suddenly their CPU usage spikes at 100% util over all cores because my vibe-coded personal site has a crypto miner package installed and i never noticed. maybe it does some data exfiltration as well just for shits and giggles. or maybe it does <insert some really dark thing here>.
“safety and security critical” applies in way more situations than people think it does within software engineering. so many mundane/boring/vibe-it-out-the-way things we do as software engineers have implicit security considerations to bear in mind (do i install package A or package B?). which is why i find the entire concept of “vibe-coding” to be nightmarish - it treats everything as a secondary consideration to convenience and laziness, including basic and boring security practices like “don’t just randomly install shit”.
human input/review/verification/validation is always required. verify the untrusted output of these systems. don’t believe the hype and don’t blindly trust them.
—
i did find the fact that google search’s assistant just parroted the crafted/fake READMEs thing particularly concerning - propagating false confidence/misplaced trust - although it’s not at all surprising given the current state of things.
genuinely feel like “classic search” and “new-fangled LLM queries” need to be split out and separated for low-level/power user vs high-level/casual questions.
at least with classic search i’m usually finding a github repo fairly quickly that i can start reading through, as an example.
at the same time, i could totally see myself scanning through a README and going “yep, sounds like what i need” and making the same mistake (i need other people checking my work too).