They do. But sometimes these bugs are less obvious and hard to spot without data. In this particular case, it was an animation bug that occurred during a tutorial of a game, whereby the animation caused the game to crash, but only on a subset of older IPhones and under certain memory conditions. So even though the build got through QA, it was tough to understand what would happen out in the wild. Well, on that morning I noticed that a large percentage of our users were starting this particular tutorial step but not completing it. And since all of our data is cohorted by both install date and build version, the engineering team knew exactly where to look to fix the issue, despite no knowledge on my part of a bug ever being introduced.
What are some good resources for engineers to pick up some advanced analysis skills (specifically to do the kind of work you do)?
Is it as simple as going back to college to do some courses in Advanced Statistics? Any particular sub branches of stats/math to focus on? Any good books you can recommend ? Thanks in advance.
PS: I work in Machine Learning,and am fairly knowledgeable in maths/stat but often have difficulty recommending the right sequence of books,courses etc, to people who want to follow a similar path - it doesn't help that I am entirely self taught - and often have to mumble vague generalities like "learn all the math you can and learn to code really well" etc. Recently I am getting questions about how someone can get good at "data science" and "behaviour analysis". Just wondering if there is a standard learning path for something like "casino science"
I think learning statistics definitely helps. Simple things like averages, stdev, probability, etc will help you. Also a healthy knowledge of SQL goes a long way.
But more importantly, its learning how to be a detective. For example, I'm not strictly a math person. I also know Python and PHP and use it to write scripts when I need to validate a theory, often working with raw logs.
You also need business acumen. Analysts are like consultants in that they work with a variety of teams and stakeholders. Everyone wants to be sure that the decision they're making is the right one, and data can help them do it. So you need an understanding of what the core issue is, how the stakeholder wants it solved, and the engineering capability to get it done in a timely fashion.
One of my favorite books is Collective Intelligence. Its a soft-intro to machine learning, and having read that and gone through those exercises along with taking a formal class on machine learning, has helped me see the importance of statistics applied at web scale.
EDIT (to the PS):
I don't think there's a standard teaching for data science. Everyone screens for it differently. If you apply for a job as a data scientist at LinkedIn for example (I did) they'll expect you to have a fairly formal CS background and will throw you questions any good software engineer should be able to solve. But they also ask you questions like "design a news feed" or "whats a good algorithm for a spam filter and how do you score it" or "how do you make an algorithm work with little data". I think "data science" is actually learning how to apply statistics at web scale problems, so my advice would be to look at things like recommenders, spam filters, classifiers, NLP, etc.
