Everyone loves a survivor, Right?
We love watching movies like Gladiator, where a muscular Russell Crowe stands tall after defeating wave after wave of enemies.
Sometimes, however, we concentrate on the fact of surviving and fixate on that and overlook everything else. This is a well-known cognitive bias that psychologists refer to as “survivorship bias.”
Put simply, survivorship bias describes our tendency to focus on the people or things that have passed some kind of selection process—whether it’s literal survival in the gladiator pits, or getting a perfect score on a standardized test—and forgetting about other important factors.
Survivorship bias explains why people often believe that cars that were made 50 years ago last longer than those made today—even though these ideas are empirically false.
The problem with falling prey to survivorship data is that it clouds your judgment and distracts you from getting to the root cause of a problem within your personal life, your team, or your product. It makes it easy to pattern-match and conflate correlation with causation.
When survivorship bias occurs, we tend to overlook the valuable lessons hidden within the failures. Also, Survivorship bias occurs when we cherry-pick success stories and overlook the broader context. This could lead you to incorrect conclusions on various things.
Researchers eliminate survivorship bias by considering both the successful subset and unsuccessful cases, ensuring a more comprehensive and unbiased analysis of the data.
Understanding survivorship bias and how it can cloud your judgment is the key to becoming a sharper, more critical thinker. That can lead to better team-making, building a more cohesive product, or making data-driven decisions like a scientist. Let’s walk through some examples!
World War II Planes
You might be familiar with this story about damaged World War II planes, for it is one of the most successful examples of survivorship bias.
The United States armed forces faced a dilemma during the war because returning bomber planes were riddled with bullet holes and they needed better ways to protect them.
The army knew they needed armor to protect their planes with bullet holes but the question was, “Where should they put it?”
When they plotted out the damage these planes were incurring, it was spread out, but largely concentrated around the tail, body, and wings.
[Source]
But Abraham Wald, a statistician at the Statistical Research Group (SRG), made a glaring observation—the military would make a terrible mistake by upgrading the armor along these sections of the plane.
Why? Because the military was only looking at the damage on returned planes.
They hadn’t factored in damage on planes that didn’t return.
Planes that didn’t return were the ones that sustained damage in ways not seen on returned planes— their engines. Unlike the body, tail, and wings, the engine was extremely vulnerable.
Once hit there, planes went down, and they didn’t make it back home to have their damage charted out.
We see the same error in perception in the business world. A product manager might hear about push notifications that worked well for a competitor so they implement something similar. But for some reason, their users aren’t receptive.
Why is that? It’s because they only heard about their competitor’s success stories, and they didn’t hear about the backlash from users who hate receiving unsolicited notifications.
To avoid making this mistake, run your own product analysis and look at it from all sides.
Extrasensory perception
Back in the 1930s, Dr. Joseph Rhine set out to test whether or not extrasensory perception (ESP) really existed. To figure this out, he tested whether someone could successfully guess the order of a shuffled deck of cards.
He originally used regular playing cards but switched to specially designed Zener cards so that participants didn’t default to guessing only the playing cards they were familiar with versus genuinely guessing.
[Source]
In his experiment, he asked 500 people to guess the order of these cards. Participants who guessed right were moved to the next round. They were asked to guess again, and those who guessed correctly were moved to the next round and so on.
Whoever was left at the end was deemed to have telepathic ability because they were able to guess correctly in every round.
As interesting as this experiment is, it was a case of survivor bias. The experiment, unintentionally, found the probability that someone, anyone, in the participant pool would guess correctly each time. It didn’t determine that a particular person had abilities.
When it comes to your product campaigns, ask yourself why one campaign performed better than others. While it might seem as though you’ve found the secret sauce to success, the success might be attributed to luck of the draw. Out of everyone in your industry, you’re the lucky one who got it right.
Not to sound too negative, but in some cases, it has more to do with luck than things you learned along the way.
Epidemiology
An unexpected place to find survivorship bias is in people diagnosed with a disease. The idea behind this is that when survival rates are discussed, they’re based on the number of months or years post-diagnosis.
This is an example of survivorship bias because patients with the best prognosis are the ones with the lowest risk, such as age, gender, and fitness level. If there are more of these patients, survival rates are disproportionately represented.
What isn’t taken into account is the number of patients who die shortly after they’ve been diagnosed. Because they die, they aren’t included in survival rate calculations, thereby inflating the survival outcome.
Cancer survival rates by stage [Source]
What does this mean for your business? Think big picture when running experiments between products. Rather than only focusing on products that make it through the growth stage and into maturity, take a close look at abandon rates.
This will give additional information into how well your product is performing over time and give you ideas on how to improve your product.
Entrepreneurship
You’re in business to succeed, and you’ve probably read your fair share of blog posts about successful entrepreneurs who’ve done well.
So what’s the problem here? Anyone thinking about becoming an entrepreneur has looked at these mega-millionaires and tried to discern a pattern to their success.
Some invested all their money while others— like Mark Zuckerberg, Bill Gates, Jack Dorsey, and Travis Kalanick— were one of the college dropouts to pursue their dream.
As simple as those options might seem, following those paths ignores the fact that for as many people who did that and succeeded, there are just as many who did that and failed.
Think about it, or better yet, Google it. You’ll be hard-pressed to easily find a list of failures.
When examining successful entrepreneurs through the lens of survivorship bias, we risk missing the insights gained from the diverse experiences of others.
Even when you try to find people who have failed, you only find those who succeeded!
As an example, wearable technology is a growing trend. The moment that clear winners emerge in the field, a rush of new entrepreneurs will come in to try and get their part of the pie.
However, just because a product idea worked for someone else doesn’t mean it’ll work for you. Take an approach unique to your business.
Hedge funds
If you’ve dabbled in hedge funds, the current market performance will give you a hint about the current situation but you know that historical trends play an important role in picking stocks to invest in.
This data will give you a clear idea of both poor performance and good performance from all companies (Stock).
You can also take a look at mutual fund performance If you are interested in that. This will give you an idea of whether to go for stock performance or mutual fund performance.
Where survivorship bias comes into play is when this approach is used as a determining factor for how funds will perform in the future. Funds can be fickle, so unless they get consistent exposure, they run the risk of becoming defunct.
Defunct hedge fund performance [Source]
A better approach is to factor in funds that haven’t performed well in the past, such as funds that were defunct after the 2008 market crash. Successful investors use all of this information to make decisions as to which funds are the best to invest in.
From a business perspective, if you run a campaign that worked really well in the past, don’t be surprised if it doesn’t perform the same way in the future. User preferences and behaviors change over time meaning you have to stay on top of these changes to roll out campaigns that resonate.
The same goes, on a higher level, for entire marketing strategies and the channels that you use to acquire customers. Platforms are transient. With significant early success one, it can be hard to see that it won’t last forever. But platform opportunities are never around for long.
Baremetrics founder Josh Pigford was the first to build an analytic solution for Stripe, but he knew that the unrivalled customer acquisition that enabled would eventually dry up.
“It’s rare to have this sort of instant access to a substantial audience with very little competition,” as Pigford says, “But it’s also something that requires really fast action. In six months the opportunity will likely be gone or much less viable.”
Churn rates
You have to be careful when analyzing your customer data. You can get the information you need in any number of ways, and while they’re not necessarily wrong, the results can be misleading.
Take the calculation of user ARR for example. You might be tempted to calculate this based on your current customers because they’re the ones actively using your product, right?
However, by only looking at current customers, you ignore the revenue of customers who churned from the following year. So instead of looking at active customers from a year ago and their ARR, you’re actually looking at a smaller pool.
[Source]
Why is this important? Well, by not looking at all users, churned and retained, you’re overestimating your revenue growth. In the above example, only looking at retained users (aka survivor bias) gives you what looks like an outstanding ARR of 112%. Once you include churned users, the rate drops to 71%. Not as outstanding, but realistic.
In some cases, it makes sense to only look at new customers but if you’re looking at your business as a whole, assess all of your customers to avoid making decisions with incomplete data.
Customer service
You’re a business, and you know that your success is tied to your relationship with your customers and how you support them. That’s why when you get negative feedback, you’re eager to dig in and figure out what went wrong.
Studies have shown that the most vocal customers are the ones who express their feelings. Everyone else will either give companies another chance or just leave.
Consider this, of the 90% of customers who don’t say anything, 78% of them leave. The 10% who complain are 90% more likely to stay. We’ve all heard the saying, “The squeaky wheel gets the grease” but let’s revise that thinking.
Instead of focusing on your unhappy customers, look at the behaviors of your happiest customers. Focus on the power users within your product because they’ll give you a model for success within your product. It’s also important to understand the difference between customer service and customer support.
Having a solid grasp of how these two similar disciplines differ from one another is vital if you’re trying to gain insight into how your existing customers feel about your product.
Beyond survivorship bias
It should be clear now that survivorship bias is everywhere we look. We usually get away with it—investing in building “the wrong” app sets you back financially but is a learning experience—but it’s still important to learn.
As David Cowan of Bessemer Venture Partners told Scientific American, “For every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”
Like Abraham Wald, who saved hundreds of soldiers’ lives based on his simple observation, you take your blinders off. This could be all you need to do to save yourself frustration and prevent your business from failing.
This is a guest post from the team over at Amplitude, a platform for product analytics. To read more on product development and data-driven decision-making, check out the Amplitude blog.