Hacker News new | past | comments | ask | show | jobs | submit login

What you typically do is include gold data in your dataset. I.e., define a HIT to be 10 sequential tasks, 3-4 of which you know the correct answer to. If a turker can't get the gold data right, their work is rejected.



An even better (and common) strategy is to reject the work that fails the gold-data test and to reject most of the rest of the work too, claiming it failed the gold-data test.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: