There's another one I've seen: climbing the wrong hill. Consider a news website doing A/B testing on headlines, and optimizing for getting more clicks on an individual headline compared to other candidates in the slot. The problem is conflating "people want to click on a headline after visiting our site" with "people want to visit our site", and optimizing for the former at the expense of the latter.

The end result is clickbait headlines that alienate visitors while they fight to grab attention.

A "wrong hill" example in our industry (recruitment software) is a focus on getting as many as possible candidates to find and apply for jobs.

If I sell widgets, then yes, I want as many as possible people to come and buy them, and I really only care about the colour of their money.

However when as a widget company I am advertising jobs, I really want only qualified candidates to apply. Otherwise I am drowning in people I just need to sort through, and reject - and when I reject them, they may stop buying my widgets.

In this situation, bounce rates (e.g. people leaving the site after viewing a job) and a lot of other common metrics are meaningless without far more context.

The equivalent in widget selling is time-wasting leads. When you are advertising a job you are looking for leads and unqualified people are just bad leads.

Having said this there is little you can do about this problem since any well advertised position will attract the unqualified since the cost is so low to just mass-blast your cv.

I have always wanted to have a fee (say $500) to apply for a position and refunded $1000 to the candidates that actually have the skills they claim and which match the job requirements. Real candidates would be rewarded for applying (easy $500) paid for by all the time wasters and dunning-kruger types. While this appeals to me I doubt this idea would be viable in the real world :)

I think what you're looking for is something like a preliminary interview round that is cheap for you to administer, combined with a financial reward for those you select to go to the next level of the process.

For incentive reasons, the budget for this must come out of the hiring company's recruiting budget. Otherwise, it's a great way for unethical folks to extract money out of the applicant pool.

An alternative way is to set up some sort of competition with prizes, and then recruit from those who do well at it.

Yes I agree. It might be possible to get around the unethical company problem by outsourcing the cash distribution to an outside organisation.

Schemes like this have been tried, though from memory it was more at the level of the candidate paying $5 to apply, no refunds.

But as you can imagine, legit companies don't really want to do it, since their reputation can be besmirched. And shyster companies/recruitment agencies anecdotally played the system for profit.

Something like this does exist though in schemes like Amway and Tupperware. With Tupperware, to join, I read that you need to shell out $2500 for your own set of the product. Then, if you don't make the grade, you don't get that money back. Not quite the same, since you always get the job.

Interesting idea. Aren't you afraid that it would attract qualified candidates who are just interested in earning easy $500 with no intentions of actually getting the job?

It is just an idea, but the idea is that you would only pay out after the interview process. If you actually get good people applying then your job would be to try and attract them to come and work with you. Sure some would just be in it for the money, but as long as you attract some serious people then it would be worth it.

This is a great little read, and really resonated with me. Like the author mentioned, there is some value in A/B testing and an incremental approach to product development. But too often organizations can become obsessed with this path forward, and continuing with the same approach becomes like trying to squeeze blood from a stone -- there are just no more significant gains to be had.

Unfortunately the company I'm at right now feels like this. The product team and CEO are obsessed with A/B testing, and don't have a more holistic approach to product development. This means that our competitors are out-innovating us in certain areas, and it's disappointing to watch.

>> "...don't have a more holistic approach to product development..."

Interestingly, the article's author works at Twitter.

Haha, if we are going to use terrible reward function analogies to measure success, why stop at A/B testing? Introduce your features as mutations to your product! Start them off with one or two customers. Those features that make your users quit in a huff will be eliminated. Those that make them recommend your app to a friend will propagate. Survival of the fittest feature!

I love this terrible idea. I can totally imagine some terrified product manager turning up the mutation rate to try to find some good set of features while people leave because they don't want a mutating product.

We sometimes call this "pivoting your business" ;-)

I really liked the animations in this article. I found them very useful in visualizing the authors points about the potential pitfalls of various product development strategies, even if they exaggerate the effects a bit. Also neat that they're SVGs instead of the usual GIFs.

Note that this has little to do with hill-climbing algorithms and much with A/B testing. Though there is an overlap in the problems both face.

I wonder if algorithmic solutions like randome restart or simulated annealing could improve the outcome of A/B testing, too.

> Bold changes require lots of resources. Maybe it’s mostly the success-bound companies who have enough resources to afford the bold changes.

And even when they do get the causality right, maybe big changes in production are likely to cause big changes in outcome - providing both greater potential reward and greater risk of losing everything.

Any sufficiently lucky contrarian is indistinguishable from genius.

Edit for clarity

Those javascript animations had my CPU running at 100% while I was reading the article. I'm not sure I like this new web... can't we just go back to GIFs for simple stuff like that?

The only reason GIFs are a better solution for you (i.e. not taking 100% of the CPU resource) is because it's about a quality GIF render. What you really need is quality of the software that powers the alternative solutions.

I often wonder if an agile development approach that only plans in the range of very short iterations could run into the same problem.

TL;DR: Incremental approach could prevent disruption benefits.

There's three to four minutes of reading in this article, possibly less if you're familiar with the concepts.

It would be great if users were required to click on the actual article and wait a certain length of time before commenting.