This is what I wanted to focus on so thanks for starting the convo. This all feels like 100% coverage = perfectly tested, no bugs possible. Nooooo, there needs to be more than that. I lately had a really good readme for a project in a heavy development phase. Basically everything I'd done, every command, every concept, got documented. That's worry about cleanup later. I did not put in every line of code, I put concepts. So when a new person got brought on and asked stuff like "well but how do I change the config?" Bam, it's in the readme. Over and over, every task I had to do, they had to at least consider or understand, so it's in the readme. Of course I did start with a quick-start "how do I use this repo" and only later did "how do I develop this repo" but still, it was all useful because it's what I needed.
It doesn't seem impossible for an LLM to go "hmmm, the way this repo passes configurations around isn't standard. I should focus more on that." But that's a level of understanding I don't think they currently have
> But that's a level of understanding I don't think they currently have
I think they do, at least in some of the cases, especially if it's something well represented in the dataset. I've been surprised sometimes by the insights it provides, and other times it's completely useless. That's one of the problems, it's unreliable, so you have to treat all info it gives you with doubt. But, anyways, at times it makes very surprising and seeming intelligent observations. It's worth at least considering it and thinking it through.
I guess I should try it before dismissing it, but I would be curious to see if it can accurately detect which things we've found workarounds for that need special attention and whatnot.
As mentioned, it's a throw of the die. It can find very obscure things that you forgot about and give you good ideas. It might come with utterly stupid ideas. You clearly need to drive these things else they will drive you and you won't like where it will take you.
It doesn't seem impossible for an LLM to go "hmmm, the way this repo passes configurations around isn't standard. I should focus more on that." But that's a level of understanding I don't think they currently have