Hacker News new | past | comments | ask | show | jobs | submit login

I am no expert in statistics or the Dunning-Kruger effect but this analysis doesn't sound correct to me. If you plot self assessment against test scores then the following will happen. If people are perfect at self assessment, then you get a straight diagonal line. The more wrong they are, the wider the line will get, in the extreme - if the self assessment is unrelated to the test result - the line will cover the entire chart. If people overestimate their performance, the line will move up, if they underestimate their performance, the line will move down. If you look at the Dunning Kruger chart, that is what you see, complicated a bit by the fact that they aggregated individual data points. At low test scores the self assessment is above the diagonal, at high test scores it is below. What matters is indeed the difference between the self assessment and the ideal diagonal, but if you don't plot individual data points but aggregate them, you have to make sure that there is a useful signal - if self assessments are random, then the median or average in each group will be 0.5 and you will get a horizontal line, but that aggregate 0.5 isn't really telling anything useful.



I'm not sure what you mean with "the wider the line will get". But here is the issue:

The least competent person cannot underestimate their relative competency. Any not exactly accurate estimate they do is an overestimate.

Correspondingly, the most competent person cannot overestimate their relative competency.

This leads to the perception of bias where there is none, except a trivial tautological one.


I made you a picture [1]. I randomly generated 100 test scores between 0 and 1, then different self assessments. Top left, self assessment matches actual score, top middle, self assessment varies uniformly by ±0.1 around the test score, top right, self assessment varies uniformly by ±0.2 around the test score. None of those have a Dunning-Kruger effect. If you aggregate data points, there will be - as you mentioned - an edge effect because the self assessment will get clipped.

In the bottom row I added a Dunning-Kruger effect, at a test score of 0.7 the self assessment is perfect, below and above that the self assessment is off by 0.5 times the distance of the test score from 0.7. Otherwise the bottom charts are the same, no random variation on the left, ±0.1 in the middle and ±0.2 on the right. You can see that the edge effect is less important as the data points are steered away from the corners.

I will admit that the original Dunning-Kruger chart could or could not show a real effect, really depends on how they aggregated the data and how noisy self assessments are. But if you have a raw data set like the one I generated, you could easily determine if there is an effect. If one could find such a data set, I would like to have a look.

[1] https://imgur.com/g4frW6p


I see what you mean. It should be possible to determine if there is an effect using the raw data from the experiment.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: