Everyone loves a survivor.
We love watching movies like Gladiator, where a muscular Russell Crowe stands tall after defeating wave after wave of enemies.
Sometimes, however, we concentrate on the fact of surviving and fixate on that, and overlook everything else. This is a well known cognitive bias that psychologists refer to as “survivorship bias.”
Put simply, survivorship bias describes our tendency to focus on the people or things that have passed some kind of selection process—whether it’s literal survival in the gladiator pits, or getting a perfect score on a standardized test—and forgetting about other important factors.
Survivorship bias explains why people often believe that cars that were made 50 years ago last longer than those made today—even though these ideas are empirically false. The problem with falling prey to survivorship data is that it clouds your judgment and distracts you from getting to the root cause of a problem within your personal life, your team, or your product. It makes it easy to pattern-match and conflate correlation with causation.
Understanding survivorship bias and how it can cloud your judgment is the key to becoming a sharper, more critical thinking. That can lead to better team-decision making, building a more cohesive product, or making data-driven decisions like a scientist. Let’s walk through some examples!
World War II Planes
You might be familiar with this story about damaged World War II planes, for it is one of the most popular examples of survivorship bias.
The United States armed forces faced a dilemma during the war, because returning bomber planes were riddled with bullet holes and they needed better ways to protect them.
The army knew they needed armor to protect their planes but the question was, “Where should they put it?”
When they plotted out the damage these planes were incurring, it was spread out, but largely concentrated around the tail, body and wings.Source]
But Abraham Wald, a statistician at the Statistical Research Group (SRG), made a glaring observation—the military would make a terrible mistake by upgrading the armor along these sections of the plane. Why? Because the military was only looking at the damage on returned planes. They hadn’t factored in damage on planes that didn’t return.
Planes that didn’t return were the ones that sustained damage in ways not seen on returned planes— their engines. Unlike the body, tail, and wings, the engine was extremely vulnerable. Once hit there, planes went down, and they didn’t make it back home to have their damage charted out.
We see the same error in perception in the business world. A product manager might hear about push notifications that worked well for a competitor so they implement something similar. But for some reason, their users aren’t receptive.
Why is that? It’s because they only heard about their competitor’s success stories, and they didn’t hear about the backlash from users who hate receiving unsolicited notifications. To avoid making this mistake, run your own product analysis and look at it from all sides.
Back in the 1930s, Dr. Joseph Rhine set out to test whether or not extrasensory perception (ESP) really existed. To figure this out, he tested whether someone could successfully guess the order of a shuffled deck of cards.
He originally used regular playing cards but switched to specially designed Zener cards so that participants didn’t default to guessing only the playing cards they were familiar with versus genuinely guessing.Source]
In his experiment, he asked 500 people to guess the order of these cards. Participants who guessed right were moved to the next round. They were asked to guess again, and those who guessed correctly were moved to the next round and so on.
Whoever was left at the end was deemed to have telepathic ability because they were able to guess correctly in every round.
As interesting as this experiment is, it was a case of survivor bias. The experiment, unintentionally, found the probability that someone, anyone, in the participant pool would guess correctly each time. It didn’t determine that a particular person had abilities.
When it comes to your product campaigns, ask yourself why one campaign performed better than others. While it might seem as though you’ve found the secret sauce to success, the success might be attributed to luck of the draw. Out of everyone in your industry, you’re the lucky one who got it right.
Not to sound too negative, but in some cases it has more to do with luck than things you learned along the way.
An unexpected place to find survivorship bias is in people diagnosed with a disease. The idea behind this is that when survival rates are discussed, they’re based on the number of months or years post-diagnosis.
This is an example of survivorship bias because patients with the best prognosis are the ones with the lowest risk, such as age, gender, and fitness level. If there are more of these patients, survival rates are disproportionately represented.
What isn’t taken into account is the number of patients who die shortly after they’ve been diagnosed. Because they die, they aren’t included in survival rate calculations, thereby inflating the survival outcome.
Cancer survival rates by stage [Source]
What does this mean for your business? Think big picture when running experiments between products. Rather than only focusing on products that make it through the growth stage and into maturity, take a close look at abandon rates.
This will give additional information into how well your product is performing over time and give you ideas on how to improve your product.
You’re in business to succeed, and you’ve probably read your fair share of blog posts about entrepreneurs who’ve done well.
So what’s the problem here? Anyone thinking about becoming an entrepreneur has looked at these mega-millionaires and tried to discern a pattern to their success. Some invested all their money while others— like Mark Zuckerberg, Jack Dorsey, and Travis Kalanick— dropped out of college to pursue their dream.
As simple as those options might seem, following those paths ignore the fact that for as many people who did that and succeeded, there are just as many who did that and failed.
Think about it, or better yet, Google it. You’ll be hard pressed to easily find a list of failures.
Even when you try to find people who have failed, you only find those who succeeded!
As an example, wearable technology is growing trend. The moment that clear winners emerge in the field, a rush of new entrepreneurs will come in to try and get their part of the pie. However, just because a product idea worked for someone else doesn’t mean it’ll work for you. Take an approach unique to your business.
If you’ve dabbled in hedge funds, you know that historical trends play an important role in picking stocks to invest in.
Where survivorship bias comes into play is when this approach is used as a determining factor for how funds will perform in the future. Funds can be fickle, so unless they get consistent exposure, they run the risk of becoming defunct.
Defunct hedge fund performance [Source]
A better approach is to factor in funds that haven’t performed well in the past, such as funds that were defunct after the 2008 market crash. Successful investors use all of this information to make decisions as to which funds are the best to invest in.
From a business perspective, if you ran a campaign that worked really well in the past, don’t be surprised if it doesn’t perform the same way in the future. User preferences and behaviors change over time meaning you have to stay on top of these changes to roll out campaigns that resonate.
The same goes, on a higher level, for entire marketing strategies and the channels that you use to acquire customers. Platforms are transient. With significant early success on one, it can be hard to see that it won’t last forever. But platform opportunities are never around for long.
Baremetrics founder Josh Pigford was the first to build an analytic solution for Stripe, but he knew that the unrivalled customer acquisition that enabled would eventually dry up: “It’s rare to have this sort of instant access to a substantial audience with very little competition,” as Pigford says, “But it’s also something that requires really fast action. In six months the opportunity will likely be gone or much less viable.”
You have to be careful when analyzing your customer data. You can get the information you need any number of ways, and while they’re not necessarily wrong, the results can be misleading.
Take the calculation of user ARR for example. You might be tempted to calculate this based on your current customers because they’re the ones actively using your product, right?
However, by only looking at current customers, you ignore the revenue of customers who churned from the following year. So instead of looking at active customers from a year ago and their ARR, you’re actually looking at a smaller pool.Source]
Why is this important? Well, by not looking at all users, churned and retained, you’re overestimating your revenue growth. In the above example, only looking at retained users (aka survivor bias) gives you what looks like outstanding ARR of 112%. Once you include churned users, the rate drops to 71%. Not as outstanding, but realistic.
In some cases, it makes sense to only look at new customers but if you’re looking at your business as a whole, assess all of your customers to avoid making decisions with incomplete data.
You’re a business, and you know that your success is tied to your relationship with your customers. That’s why when you get negative feedback, you’re eager to dig in and figure out what went wrong.
Studies have shown that the most vocal customers are the ones who’ll express their feelings. Everyone else will either give companies another chance or just leave.
Consider this, of the 90% of customers who don’t say anything, 78% of them leave. The 10% who complain are 90% more likely to stay. We’ve all heard the saying, “the squeaky wheel gets the grease” but let’s revise that thinking.
Instead of focusing on your unhappy customers, look at the behaviors of your happiest customers. Focus on the power users within your product because they’ll give you a model for success within your product.
Beyond survivorship bias
It should be clear now that survivorship bias is everywhere we look. We usually get away with it—investing in building “the wrong” app sets you back financially but is a learning experience—but it’s still important to learn.
As David Cowan of Bessemer Venture Partners told Scientific American, “For every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”
Like Abraham Wald, who saved hundreds of soldiers lives based on his simple observation, you take your blinders off. This could be all you need to do to save yourself frustration and prevent your business from failing.
This is a guest post from the team over at Amplitude, a platform for product analytics. To read more on product development and data-driven decision-making, check out the Amplitude blog.
P.S. If you liked this article, you should subscribe to our newsletter. We’ll email you a daily blog post with actionable and unconventional advice on how to work better.