Hacker News new | past | comments | ask | show | jobs | submit login

Well, I wouldn't say "equivalent" (and I wouldn't say it's implied in the article), but it isn't very far from that. And meaning of "very far" is relative. I mean, f(x,y) ~ x obviously isn't equivalent to x ~ x, but the importance of pointing this out depends on how likely people are to mistake it for x ~ y, and how likely they are to skip that f() is DEFINED by authors and not found in the nature. Same with "misleading": your perspective on if something is misleading obviously depends on where it leads you. But my evaluation of 2 questions above are "very likely" and "it does lead majority of people to the wrong conclusions", so the whole sentiments stands true. I'll get back to it in a moment.

Anyway, my (lesser) point is that a study like that (even in 1999) must contain at least one plot of [actual score] vs [self-assessment score], or it is worthless, and I stand by it. (The original paper doesn't contain such plot.) My more general point is that what is accepted as "sufficient" data provided in overwhelming majority of research is laughable, and I really want people to stop tolerating that.

(I don't want to delve into discussing the particular paper, I'm purposefully trying to keep all my point as general as possible. Some problems are actually addressed in DK-1999, but the whole study makes you smirk at a circus social psychology experiments are. Here's a nice quote for you (being tested is "ability to recognize what's funny"): "To assess joke quality, we contacted several professional comedians … and asked them to rate each joke on a scale ranging from 1 (not at all funny) to 11 (very funny). Eight comedians responded to our request …, an analysis of interrater correlations found that one (and only one) comedian's ratings failed to correlate positively with the others (mean r = -.09). We thus excluded this comedian's ratings in our calculation of the humor value of each joke.")

Now, back to the f(x, y) ~ x. Let's imagine f(x, y) = C. E.g., every respondent evaluated himself as "average". Is it true that less knowledgeable people turned out to overestimate themselves, and more knowledgeable people turned out to underestimate themselves? Well, yeah, I guess. Does x correlate with y? No. So, what's more valuable here, to plot [x ~ y] or [f(x, y) ~ x]? Moreover, this is obviously different from the case where y (the self-assessment score) is uniformly distributed and doesn't correlate with x, yet it will yield the same line-plot as the above, if we construct them like D&K did. And that's skipping the part that there are several viable ways to define "self-assessment of performance" as f(x,y). I think, if I'll try hard enough I would be able to "prove" almost any hypothesis about self-assessment using this method. Of course, I won't really prove anything, but the plots will be convincing enough, and the "results" will be much more demonstrative than the original data is, same as probably is with DK-1999 (but that's impossible to tell without seeing actual source data of DK-1999).

TL;DR: It all boils down to the old adage about "3 kinds of lies". Is the result being discussed an example of auto-correlation? Yes, it is. Is auto-correlation necessarily bad? Not really. Is it important and can it obstruct interpretation in the particular example of DK-1999 paper? It's subjective and hasn't been measured, but my evaluation is — absolutely.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: