Harvard Business School - Professor Michael Luca - The Power of Experiments
“ Over the past 50 years we’ve seen a growing body of evidence that indicates that we don’t know in advance what will work. Our intuition is flawed. Experiments are one way to check our intuition, making sure that we’re removing biases from our decisions where possible”.
You might also enjoy these articles:
Introduction
Is data enough?
Or are there times when data can mislead us? These are the questions that all leaders and decision-makers need to be asking.
And if we’re not asking the right questions — or conducting the right experiments — is data worthless?
There can often be a gap between the experiments that are being performed, and the decisions that are being put in place. Managerial decisions need to link back to product decisions.
Recently I spoke with author, educator, and researcher Michael Luca. Luca is a Professor at Harvard Business School, who has been published in the Wall Street Journal, The Atlantic and WIRED.
We discuss Professor Luca’s book The Power of Experiments: Decision Making in a Data Driven World, including case studies from eBay, Airbnb, and Alibaba.
In this article we discuss:
A grounding perspective on data
Humble leaders are the best leaders
Understanding organisational dynamics
eBay’s $50M advertising mistake
Airbnb’s moral dilemma
Deep discounts at Alibaba
1. A grounding perspective on data
It’s no longer an esoteric set of social scientists and medical researchers that are performing experiments, but also companies and governments too.
Experimentation is complementary to data analysis and other forms of customer research to guide day-to-day decision-making.
Experimentation has become analogous with businesses. Organisations have data, metrics that they’re measuring and a high-volume of customers that can be randomised easily and cheaply.
If you’re tossing up between two different solutions on a platform, why wouldn’t you experiment? There’s no longer a need to guess. Expensive failure does not have to be an option.
“Over the last 50 years or so we've seen a growing body of evidence that it's hard to understand exactly what will work. Our intuition can often be flawed. And experiments and other types of data are one way to help check our intuition and help to make sure that we're removing biases from our decisions where possible”.
As a leader, if you’re not thinking about experimentation, you’re introducing blind spots.
Should all leaders be trying new things in their organisation? YES.
Should all leaders be gathering more information to understand what will and won’t work? YES.
Should all leaders have a sense of humility? YES
2. Humble leaders are the best leaders
One of the cornerstones of a strong experimentation culture is humility. Leaders need to be willing to engage in discussion of data and evidence.
Humility is important as it helps us to understand when we might have been wrong. It provides us with the agility to try different options, acknowledging when things didn’t work so well.
If humility is valued in an organisation, it can set the company on a path for long-term success – on a completely different trajectory. The power of experimentation can be fully harnessed.
“You can't just reward victories. You must redefine what it means to be successful as an organisation. Trying and failing, trying, and failing early, and then moving on to the right thing, should be rewarded rather than punished”.
There are some things that you need to get right in the beginning - experiments need to be tightly anchored back to organisational strategies, objectives, and metrics.
Understanding the different roles and purposes that experimentation can play in your organisation will help leaders to get more out of experimentation.
3. Understanding organisational dynamics
Leaders need to take the time to understand the organisational dynamics at play.
Think about these three questions.
1. What are the organisational dynamics?
2. What are the barriers to evidence seeking?
3. How might we overcome barriers to evidence seeking?
Take the time to zoom out and understand why you feel like experimentation may be helpful, but other leaders don’t agree.
Thinking about the source of barriers to experimentation, some may include:
Experimentation perceived as costly – time, resource, and budget
People may not want to find new information to disconfirm their beliefs
People may not be incentivised to experiment/learn
People don’t know what to do with data and insights from experiments
Unwillingness to change – have the answers and know the path forward
These are all important barriers to experimentation.
Once you start thinking about these broader organisational issues it will help you point out what experiments you need to run, who needs to run them and where experiments need to be performed.
If you’re enjoying this article, listen to the full podcast conversation on Experimentation Masters
4. eBay’s $50M advertising mistake
eBay is no stranger to experimentation and using customer data to increase competitive advantage.
For several years, eBay marketers had been working to increase traffic to the website using paid search advertising to drive customer purchases.
On surface level, this seemed like a sound strategy and good investment. Customers who clicked on the search ads ended up buying stuff on eBay. That’s a win, right?
Head of eBay’s Economics Group, Steve Tadelis, wasn’t convinced. The company was paying to advertise on Google when users searched for “ebay” and other generic brand-related keywords.
The experiment:
The hypothesis – these users were already prepared to shop on eBay. Users would find their way to the website even without paid advertising.
However, when eBay’s economics research team conducted a series of experiments to find out the exact financial returns of eBay’s paid search investment, they found that much of the budget eBay spent on search ads each year ($50M) was going to waste.
eBay economists Tom Blake, Steve Tadelis and Chris Nosko performed a series of experiments where eBay search ads were systematically turned on and off in different markets. They tracked traffic coming from paid search and organic search.
The results:
When paid ads were turned off it resulted in a big spike in organic traffic.
Why? Loyal, repeat customers had been clicking on paid ads as there was no reason to scroll further down the page to click on organic search results. These customers would’ve visited eBay anyway.
The research did find that using paid search for items that are less commonly associated with eBay could result in a boost to sales. For example, a “used gibson les paul guitar”. Additionally, paid search is more effective for inexperienced eBay buyers or less informed prospective customers.
How did eBay course correct after these learnings?
“Leaders thought about other areas where they might then start to experiment, and began running experiments in different markets,” says Luca. “Leaders realised that not only is experimentation going to help them discover new things, but that it’s going to help them test existing products and services to determine if they’ve been on the right track with the things they’re doing.”
Practical takeaways:
Experiments should be used to find answers to important business questions.
Without experiments, it’s easy to fall into the trap of relying on correlation for decision-making. In eBay’s case, the correlation that advertising leads to more revenue. There’s always the risk the correlations may not be causal.
It’s almost impossible to understand if you’re on the right track or not – that goes for both new plans and existing strategies.
eBay was able to shift their Marketing and advertising strategy by understanding the causal relationships of their actions and with experiments.
5. Airbnb’s moral dilemma
While on many online marketplaces the anonymity of buyers and sellers is maintained, Airbnb requires a personal profile.
Hosts are at liberty to accept and reject prospective renters without having to explain why.
Luca was curious: even with Airbnb’s outward success, was Airbnb experimenting with the right objectives in mind?
He set out to perform an experiment to answer the question: Was discrimination against guests a real problem on Airbnb?
“We had come in and started looking at not just the productivity of the platform, but the inclusivity of the platform,” says Luca. “And with my co-researchers Ben Adam and Dan Sversky we ran an experiment where we tested for racial discrimination on the platform.”
The experiment
Posing as guests, 6,400 rental inquiries were sent to hosts across the United States. All of the enquiries were identical, except for one characteristic – half were from common Anglo-Saxon names and half were from common African American names.
Findings indicated that there was widespread discrimination against African Americans on Airbnb. Inquiries from guests with distinctively African American names were 16% less likely to receive a “yes” from hosts than those with Anglo-Saxon names.
Discrimination was evident across a range of different neighbourhoods, listing types, price points, property types and host types.
Practical takeaways:
All organisations are trying to optimise for more growth and revenue.
Airbnb had been optimising too narrow on their goal of growing corporate profits without considering the unintended social consequences and risks of their decisions.
Experimentation does not operate in a vacuum. It’s always important to be thinking about the upstream and downstream impacts of your experiments. Often, experiments can produce unintended consequences.
In this case, Airbnb’s product design decisions had facilitated discrimination because of a linear focus on short-run growth objectives.
“Think of things occurring outside of your direct experiment, to bring more things inside your experiment.”
6. Alibaba’s deep discounts
Not just any experiment will do. Experiments need to be linked back to managerial decision-making.
Experiments must move you from experimentation results to product decisions.
The experiment:
One area where Alibaba experimented was the discounts provided to customers. The business wanted to understand which situations discounts would be most beneficial in driving user engagement and retention.
Through data analysis, Alibaba noticed that customers would put items in their shopping cart without purchasing them. Many customers didn’t purchase these products at all.
Alibaba had the idea of allowing sellers to provide a “deep discount” on promoted items that were left in a shopping cart for over 24 hours.
Alibaba conducted a randomised, controlled experiment to more than one million customers. Customers in the control group received no discount promotions and customers in the treatment group received a discount coupon.
The results:
The results showed that customers were more likely to buy a product if it was discounted in the shopping cart than if it was not discounted.
However, there was a big but … The experimentation team discovered that over time customer basket value was not increasing, rather decreasing.
Customers who received a discount coupon started adding more items to their basket and letting them sit there in the hope of triggering further discounts.
Long-term, this was bad news for Alibaba as it resulted in revenue cannabilisation. Alibaba was conditioning customers to hold out and wait for discounts rather than paying full list price.
Based on these experimentation results Alibaba executives decided to not proceed with the discount program.
“Maybe you learn that it's not that discounts aren't effective. It's the fact that we didn't design the discounts in a way that was going to drive the right behaviour.”
Practical takeaways:
Experimentation starts with asking the right questions. In this case, Alibaba did not ask the right questions upfront.
The questions Alibaba asked were “how much should we discount” and “how regularly should we discount”. These were the wrong questions to ask.
Questions that the Alibaba team could’ve asked are “How might we design our monetisation model” or “How might we design a pricing and discount program”. These are two very different questions that drive two very different strategic and commercial outcomes.
Asking better questions upfront yields better business outcomes in the long-term
Summary - When it comes to making good decisions, data is only half the battle
If there’s anything I’ve learned from Luca’s perspective on experimentation, it’s that evidence and data are only half the battle when it comes to smarter leadership.
Three key takeaways include:
Try not to mistake correlation for causation
Understand the upstream and downstream impacts of your experiments (and decisions)
Be a better product designer by asking the right questions upfront.
Achieving success with experimentation requires a tight alignment on business strategy, objectives and metrics, a healthy dose of organisational humility and a desire from leaders in the first instance to be evidence seeking.
By learning from the experimentation missteps of the world’s leading online platforms — while simultaneously recognising that missteps are a necessary and inevitable part of organisational learning — it’s possible to steer your organisation in ways that propel profits, drive growth, and deliver more value for your customers.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.