This article features a study claiming 25% productivity gains form co-pilot. I wonder what a similar study would say about developers with and without access to internet search. In the study, it seems, as I expected, less experienced people found more value from it. My experience has been AI tools are more helpful when I don't know the domain, and then their benefit declines as I familiarize.
I also looked at the study and noticed a few aspects that were surprising:
(a) some of the 95% CIs crossed zero, meaning no benefit is a possible interpretation in figures 6 and 7
(b) did anyone account for what happens when two workers are in different experimental groups and sit next to each other? I imagine it was likely common people in the experimental group to use co-pilot queries for their friends.
(c) for experienced workers, the mean value is actually negative (lol) in the unweighted data in Figure 7
There are a lot of other subtleties in the interpretation of this paper.
"...developers who were less productive before the
experiment are significantly more likely to accept any given suggestion...."
Yeah out of all the studies I’ve seen about actual productivity gains from LLMs the median estimate is probably in the low single digits (possibly including 0). Not exactly earth shattering yet.
I also looked at the study and noticed a few aspects that were surprising:
(a) some of the 95% CIs crossed zero, meaning no benefit is a possible interpretation in figures 6 and 7
(b) did anyone account for what happens when two workers are in different experimental groups and sit next to each other? I imagine it was likely common people in the experimental group to use co-pilot queries for their friends.
(c) for experienced workers, the mean value is actually negative (lol) in the unweighted data in Figure 7
There are a lot of other subtleties in the interpretation of this paper.
"...developers who were less productive before the experiment are significantly more likely to accept any given suggestion...."
...curious lol
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566