I went through the report, and they reference other studies that consider my points, but they end up only measuring incident numbers, don't specifically say how comparable were the driving conditions (iow whether they are representative). If anything, they could have removed incidents by human drivers which did not happen within their operating constraints, but I did not see that mentioned.

But that data isn’t available to them. They are (iiuc) simply looking at the full set of insurance data and their own set. There simply is no driver-level data which would allow statistically relevant conclusions.

Up to now their data for their own cars is still not able to strongly constrain the results. You wouldn’t be able to tell effects of sensor degradation.

But let me ask this instead: What kind of analysis would you like to see?

The report specifically mentions a number of other studies and claims it improves on their methodology, and seems to want to have their methodology become the "gold" standard of measuring SDV vs HDV performance:

  "This methodology establishes a foundation for future research into the safety impact potential of ADS and offers a framework for assessment as these systems continue to scale and develop."

Or:

  "By analyzing an unprecedented volume of autonomous miles, introducing a novel benchmark, and utilizing third-party liability insurance claims, we aim to set a new standard for ADS safety evaluation."

Which leads me to worry that they are pushing for this to be considered comparative, and they want policy to be influenced by their claims. And they do:

  "Our findings have the potential to significantly impact policy decisions, insurance practices, and public acceptance of ADS technology, contributing to the broader societal dialogue on the future of autonomous transportation."

So I would like for them to compare to median non-impaired human driver with a car in good order (both in time-to-destination and safety), aggregate those over time of day, week and year (basically ensure identical driving conditions between comparison miles), adding claim dollar amount as weight for how severe the incident was. I believe that would be a great start.

With them being Google and already tracking most Android phones as they go through traffic (they use this for traffic conditions), they do have a very rich data set they could compare to as well, but excluding that, they could exclude insurance claims which have happened in conditions they don't operate in (claims will always have time of event and location).