Is it even possible in principle for an LLM to produce a confidence interval given that in a lot of cases the input is essentially untrusted?
What comes to mind is - I consider myself an intelligent being capable of recognising my limits - but if you put my brain in a vat and taught me a new field of science, I could quite easily make claims about it that were completely incorrect if your teaching was incorrect because I have no actual real world experience to match it up to.
Right, and that's why "years of experience" matters in humans. You will be giving incorrect answers, but as long as you get feedback, you will improve, or at least calibrate your confidence meter.
This is not the case with current model - they are forever stuck at junior level, and they won't improve no matter how much you correct them.
I know humans like that too. I don't ask them questions that I need good answers too.
What comes to mind is - I consider myself an intelligent being capable of recognising my limits - but if you put my brain in a vat and taught me a new field of science, I could quite easily make claims about it that were completely incorrect if your teaching was incorrect because I have no actual real world experience to match it up to.