>Overall, the two combinations with the highest multivariate validity and utility for predicting job performance were GMA plus an integrity test (mean validity of .78) and GMA plus a structured interview (mean validity of .76). Similar results were obtained for these two combinations in the prediction of performance in job training programs. A further advantage of these two combinations is that they can be used for both entry level hiring and selection of experienced job applicants.
"Accomplishments" could probably be called a combination of "Biographical Data" and "Reference Checks" as described in that paper. Biodata correlates highly with GMA tests, so it's a lot easier to just give a GMA test then pour over descriptions of of past experiences. Reference checks are generally not useful today because of the fear of lawsuits associated with giving a reference that costs someone a job.
It seems that a general programming test combined with a structured interview has data showing it's a good predictor for on-the-job performance.
Always appreciate seeing a highly cited and influential paper, thank you for sharing that. It seems to make a good case that IQ (or GMA or similar) combined with one or two simple tests can have some success as a tool for choosing candidates.
The issue I would have is it doesn't seem to preclude the possibility that placing more emphasis on specific accomplishments and other software development contributions, is as I suspect, an even better approach.
You mention biographical data and reference checks. I dug just a little further into related papers to get a better idea of what the authors meant by this. From what I can tell those concepts are different things from what we might guess, and wouldn't serve as a proxy to evaluate my conjecture (for example, on the meaning of biographical data: https://www.biz.uiowa.edu/faculty/fschmidt/meta-analysis/rot...).
Speaking of conjecture, I realize that's what I'm offering here. Whereas they have strong data to say, here's a couple of approaches that can be effective.
Anecdotally, I still believe better candidates could be chosen by closely evaluating the specific work a developer has done, what role they played on the teams, and the impact and value of the work directly connected to their contributions. Of course these can't work just by listing Git repos, the interview process has to explore and validate the history. Combining this with the interview process can allow great insights into a candidates thinking, because it allows discussions about real world, non-trivial work and problem solving.
Not specifically tailored to programmers, but I find this paper relevant with respect to hiring practices:
The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2853669
An excerpt:
>Overall, the two combinations with the highest multivariate validity and utility for predicting job performance were GMA plus an integrity test (mean validity of .78) and GMA plus a structured interview (mean validity of .76). Similar results were obtained for these two combinations in the prediction of performance in job training programs. A further advantage of these two combinations is that they can be used for both entry level hiring and selection of experienced job applicants.
"Accomplishments" could probably be called a combination of "Biographical Data" and "Reference Checks" as described in that paper. Biodata correlates highly with GMA tests, so it's a lot easier to just give a GMA test then pour over descriptions of of past experiences. Reference checks are generally not useful today because of the fear of lawsuits associated with giving a reference that costs someone a job.
It seems that a general programming test combined with a structured interview has data showing it's a good predictor for on-the-job performance.