Telemetry isn't an issue. If anything, it can be fully disabled. And yours is done right - all the data sent is properly described and observable. (Although it would be cool if there'd be a helper "let me decide later" mode when I can let browser collect data for a short while, get notified to review it, and then decide whenever it contains anything I consider sensitive or not...)
Maybe this is silly and irrational, but... While it's all your work (and thank you for this!) and decision, it just somehow doesn't feel right that an locally-installed software - no matter how experimental - is ephemeral, almost like it's not a software but a service. With normal AMO addons - even the experimental ones - I can browse the version history and roll back if something's not right. Test Pilot feels like end-user is completely out of control of whatever happens (unless they don't participate and just install from Git repos).
You're right. We keep control over the experiments so that we know the consistency of the data we're measuring.
A problem with using our full release channel Telemetry is that so many users have so many variables it's hard to compare apples to apples. E.g., was their performance issue caused by a setting, an add-on, the site they were on, or Firefox itself?
Test Pilot experiments give us some controls and parameters on the experience so we have a clearer idea of what's causing what.
Telemetry isn't an issue. If anything, it can be fully disabled. And yours is done right - all the data sent is properly described and observable. (Although it would be cool if there'd be a helper "let me decide later" mode when I can let browser collect data for a short while, get notified to review it, and then decide whenever it contains anything I consider sensitive or not...)
Maybe this is silly and irrational, but... While it's all your work (and thank you for this!) and decision, it just somehow doesn't feel right that an locally-installed software - no matter how experimental - is ephemeral, almost like it's not a software but a service. With normal AMO addons - even the experimental ones - I can browse the version history and roll back if something's not right. Test Pilot feels like end-user is completely out of control of whatever happens (unless they don't participate and just install from Git repos).