Nitpick: I'd ask for NITPICK at the end of output instead of the start. The model should be in a better place to make that decision there.

I find it ironic how capable LLMs have become, but they're still struggling with things like this. Great reminder that it's still text prediction at its core.

I find it more helpful to keep in mind the autoregressive nature rather than the prediction. 'Text prediction' brings to mind that it is guessing what word would follow another word, but it is doing so much more than that. 'Autoregressive' brings to mind that it is using its previously generated output to create new output every time it comes up with another token. In that case you immediately understand that it would have to make the determination of severity after it has generated the description of the issue.

I mean, think of LLM output as unfiltered thinking. If you were to make that determination would you make it before you had thought it through?

We tried this too, better but not good enough. It also often labeled critical issues as nitpicks, which is unacceptable in our context.

Then some of the nitpicks were actually valid?

What about PossiblyWrong, then?

My thinking too. Or similarly including some coding style and commenting style rules in the repository so they become part of the code base and get considered by the LLM

My boss can't stop nitpicking, and I've started telling him that "if your comment doesn't affect the output I'm ignoring it".

That is such an over-simplistic criteria! Proof by the absurd: pass your code through an obfuscator / simplifier and the output won't be affected.

Ok?

So if the boss's nitpick is "you must pass your code thru an indicator that doesn't affect the output", I'm going to ignore it. I don't understand what point you're trying to make.