The article is disappointingly misleading and buries one of the key details
"Amazon’s lawyers rejected the overt addition of contribution profit into the algorithm...
They turned to the metrics Amazon uses to test the algorithm’s success in reaching certain business objectives, said the people who worked on the project.
When engineers test new variables in the algorithm, Amazon gauges the results against a handful of metrics. Among these metrics: unit sales of listings and the dollar value of orders for listings. Positive results for the metrics correlated with high customer satisfaction and helped determine the ranking of listings a search presented to the customer.
Now, engineers would need to consider another metric—improving profitability—said the people who worked on the project. Variables added to the algorithm would essentially become what one of these people called “proxies” for profit: The variables would correlate with improved profitability for Amazon, but an outside observer might not be able to tell that. The variables could also inherently be good for the customer."
Tldr: some people in Amazon wanted to give a boost to Amazon-products. The search team fought them vociferously and refused to budge. The lawyers came out against it as well. Amazon's internal A/B testing framework measures a number of metrics, including both revenue and profit, when determining whether a specific feature/change should be deployed.
The fact that the A/B testing framework measures the profit impact of any change, is hardly earth shattering. This is one of the core features that any A/B testing framework attempts to accomplish.
This also doesn't tell you anything about what the search algorithm is actually doing. An A/B testing framework can only help you evaluate the relative effectiveness of different algorithms. It doesn't actually create/influence the algorithm in any way. As WSJ themselves reported, Amazon's search algorithm does not take profitability as an input.
WSJ has done a fantastic job in unearthing this very interesting internal-debate that's happening in Amazon. But they have reported it in a way that is very misleading and gives laypeople the impression that they have a smoking gun. In reality, the only thing they have produced is the fact that Amazon's A/B testing framework is profit-aware.
Thank you. I was struggling to understand what the article was trying to tell me or why it was so lengthy. It feels like news has evolved to buzzword/sensationalist headline + enough content/data (even if irrelevant) to provide a sense of comprehensiveness/substance.
"Amazon’s lawyers rejected the overt addition of contribution profit into the algorithm...
They turned to the metrics Amazon uses to test the algorithm’s success in reaching certain business objectives, said the people who worked on the project.
When engineers test new variables in the algorithm, Amazon gauges the results against a handful of metrics. Among these metrics: unit sales of listings and the dollar value of orders for listings. Positive results for the metrics correlated with high customer satisfaction and helped determine the ranking of listings a search presented to the customer.
Now, engineers would need to consider another metric—improving profitability—said the people who worked on the project. Variables added to the algorithm would essentially become what one of these people called “proxies” for profit: The variables would correlate with improved profitability for Amazon, but an outside observer might not be able to tell that. The variables could also inherently be good for the customer."
Tldr: some people in Amazon wanted to give a boost to Amazon-products. The search team fought them vociferously and refused to budge. The lawyers came out against it as well. Amazon's internal A/B testing framework measures a number of metrics, including both revenue and profit, when determining whether a specific feature/change should be deployed.
The fact that the A/B testing framework measures the profit impact of any change, is hardly earth shattering. This is one of the core features that any A/B testing framework attempts to accomplish.
This also doesn't tell you anything about what the search algorithm is actually doing. An A/B testing framework can only help you evaluate the relative effectiveness of different algorithms. It doesn't actually create/influence the algorithm in any way. As WSJ themselves reported, Amazon's search algorithm does not take profitability as an input.
WSJ has done a fantastic job in unearthing this very interesting internal-debate that's happening in Amazon. But they have reported it in a way that is very misleading and gives laypeople the impression that they have a smoking gun. In reality, the only thing they have produced is the fact that Amazon's A/B testing framework is profit-aware.