First of all, thanks for the wonderful work, and I hope there's much more to come from your team! In fact I'm really pleased one of the authors came down to HN to comment.

I think the scariest part of AI security is when the program itself becomes unfathomable. By that I mean, we can't just look at the source code and go "Ah! There's your problem". Now, your paper assumes a static reward function, but we can imagine the benefits of an AI that could dynamically change its reward function, or even its own source code.

In fact, the most powerful tool I can think of to train a multi-purpose agent is through evolutionary methods, and genetic algorithms. Take for example the bigger ideas behind https://arxiv.org/abs/1606.02580 [Convolution by Evolution: Differentiable Pattern Producing Networks] and http://arxiv.org/abs/1302.4519 [A Genetic Algorithm for Power-Aware Virtual Machine Allocation in Private Cloud], and determining the fitness of agents by the global accuracy on a large number of broad ML tasks. But I digress...

Given enough computing power and time, these have the possibility of ending in an "outbreak-style" scenario. [This exercise is left to the reader]. And the way AI ideas and methods are so rapidly disseminated and readily available, it's safe to imagine that it could happen in a relatively short time span.

Here's my question: I know you're with Google Brain, but do you know if OpenAI is actively researching these avenues of "self-determined" agents? For their first security-related article, I was expecting security measures along the lines of: safety guidelines for AI researchers, containment and exclusion from the Internet, shutdown protocols for the Internet backbone, etc. I get the impression some of these issues might rear their ugly heads before our cleaning robots become cumbersome.

P.S. Looking at your CV, it's funny to see that you once interned at Environment Canada. I'm also working there presently, during which time I can perfect my knowledge in ML to eventually transition careers. Small world...

Edit: Grammar.