Overview of Business Experimentation
Experimentation is the quantification of doubt. Where doubt is present, risk is present. Deliver more of your projects successfully, using experimentation to reduce business risk.
Introduction
Developing new products and services in large enterprises can be challenging.
Project delivery typically takes months or years, with capital investment running into many millions of dollars.
Even with inordinate amounts of planning and governance, the likelihood of projects underperforming is still very high.
This delivery model is not geared to discovery and exploration. Many small, iterative in-market tests are required to unearth what customers value, before proceeding to a mass-market launch.
Using a big-bang approach to execution carries significant risk. Product and market assumptions are only validated with customers when the new product is launched in the market.
Businesses need to approach development of new products and services differently.
Organisations require new ways of working and tools which support iterative product development, whereby new products can be discovered through fast, low-cost experimentation.
Overview of business experimentation
Business leadership is changing
Nowadays, the answers to many business problems are unknown. Customers, competitors and market conditions can be highly unpredictable.
Business leaders are required to operate in a world with insufficient data to make decisions. As such, decision-making can often rely on past experiences, intuition and legacy thinking.
The challenge with using this decision-making approach is that new developments and innovations fly in the face of existing experience and wisdom.
Boeing would never launch a new jet engine to market without conducting extensive testing and experimentation based on sound engineering and scientific principles. The stakes are too high.
Yet, many companies still launch new business models, products and features without a thorough testing and validation process with customers.
If businesses embark on a disciplined experimentation process prior to launching new ideas, organ rejection from customers can be discovered before launch, at low cost.
“Part of establishing a culture of data-driven experimentation, is having the humility of not knowing the right answer, or maybe not knowing at all”
Learning is a natural part of the entrepreneurial process.
The role of the business leader is transitioning from business manager to CEO , the Chief Experimentation Officer.
Experimentation creates more options
Through a disciplined process of discovery and exploration, experimentation helps to create more options.
By running many fast and low-cost experiments, it’s possible to adjust effort and directionality after discovering what does and, doesn’t work.
Experimentation allows businesses to try new strategies and ideas at small scale – a small audience subset, one geography, variants of ideas.
Strategies and hypotheses can then be refined on the run, rather than waiting for quarterly or annual planning cycles.
“Experimentation forces a mindset shift to think in terms of hypotheses, rather than ideas”
It’s impossible to know in advance which hypotheses will be correct. Consumer reactions to new concepts are always surprising – both positively and negatively.
Experimentation enables business leaders to predict with a higher level of confidence how their new idea will perform at scale.
Make better decisions & business investments
Experimentation helps to point you in the right direction when:
Answers aren’t obvious
People have conflicting opinions
There’s uncertainty about the value or merit of an idea
Experimentation reduces wasted effort – capital, time and resources.
Many organisations spend a considerable amount of time and resources developing solutions that generate little business impact.
An experimentation led approach helps businesses to weed out initiatives that have no merit, quickly understanding which ideas show promise.
Experimentation facilitates gathering of high-quality, relevant, real-world data from customers.
“Business leaders are able to connect the dots between strategy, risk and investment faster to make better quality decisions”
When performed in a systematic and industrialised manner, experimentation enables organisations to innovate more efficiently, build new products and services that meet the needs of their customers, and develop solutions to previously unanswered questions.
If you want to be a more effective and successful business leader, experimentation is the answer.
Experimentation is still challenging for many businesses
Many businesses still have difficulty executing experiments.
“The process of running experiments is relatively easy, however, organisational constraints can make executing experiments hard”
Existing business processes, governance frameworks and funding models are typically not geared for fast, low-cost experimentation. These operating models support predictability and certainty, rather than iterative learning approaches.
Experiments can also struggle for prioritisation when competing against other high priority business projects or customer campaigns.
To succeed in large organisations, experimentation requires dedicated funding, resourcing, processes and tools and infrastructure.
Make it easy for teams to run experiments and analyse results.
What are the challenges?
Human beings regularly suffer from optimism bias. We overestimate how good our ideas are, thinking that every opportunity will succeed.
We completely underestimate the likelihood of negative events occurring, where our ideas are actually bad.
However, failure is omnipresent. It’s around us every day, we just choose to ignore it, thinking that we’re averse from failure happening to us too. It only happens to everyone else right?
“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with the experiment, it’s wrong” – Richard Feynman
Market failure statistics are consistent, year on year:
A study by AC Nielsen suggested that up to 85% of 24,543 new consumer goods products launched to market in one year failed. The products either underperformed or failed relative to business case expectations
In the United States, 20% of small businesses fail in Year One, with a further 30% failing in Year Two. Only 50% of small businesses make it beyond Year Five
Harvard Business School lecturer Shikhar Ghosh published a Wall Street Journal article suggesting that 75% of venture-backed companies never provide a ROI to investors
At Microsoft, only 33% of experiments generate positive commercial outcomes, with two thirds yielding a neutral or negative outcome
The Startup Genome in their 2019 Global Startup Ecosystem Report suggest that only 1 in 12 entrepreneurs succeed
The odds are against us before we even start. We just choose not to acknowledge it.
What are the complicating factors?
So, the next obvious question, why do so many new product launches underperform in the market?
Delivering new products and services is thought to work as a one-way production line.
An idea is input at the beginning of the conveyor belt, transformation activities occur along the production line to “manufacture” a new offering, with a pot of money spit out the other end.
Customers and markets are unpredictable and, developing new products and services is a non-linear event. Remember the famous innovation squiggle?
One of the primary reasons that so many new ventures fail is that businesses skip the customer validation phase in the “manufacturing” process, jumping straight from “I’ve got a great idea” to “Let’s start building”.
“Failing to understand market demand, customer behaviours and preferences prior to launching a new product or service leads to massive organisational blind spots”
The longer these blind spots persist, the greater the inherent business risk in the project delivery process.
Ultimately, if the blind spots and assumptions remain unchecked, things can end up terminal … you realise that customers aren’t interested in your offer and you’ve wasted significant time, capital and resources building something that customers don’t want.
When is it best to run experiments?
There are two types of questions in business, those that you already know the answer, and those that you don’t know the answer.
For the questions that you already know the answer (system upgrades, continuous improvement, regulatory, compliance) you should work to understand cost-benefit and risk, prepare a business case, and get moving.
You’re not asking, “should we do it”, you just need to do it. Business experimentation is not as beneficial in these types of scenarios.
For business questions where you don’t know the answer – the “should we do it” or “how do we do it” questions – experimentation is a powerful way gather data to inform better quality decision making.
The latter scenario represents the majority of circumstances in the modern business.
What is experimentation?
Experimentation is underpinned by the scientific method, being more science than art.
It’s an iterative learning approach, shifting between inductive thinking (theory, guess) and deductive thinking (data, facts) to test business hypotheses.
“Experimentation is the quantification of doubt. Where doubt is present, risk is present”
It’s a continuous cycle of learning and exploration – hypothesise, experiment, learnings, repeat.
In the same way that the scientific method helped humanity accelerate many important developments, a scientific method of experimentation can also help business to solve complex problems much faster.
Experimentation - hypothesise, experiment, learn, repeat
An experiment represents an individual learning event with customers. It’s an opportunity to learn something new, something that you previously didn’t know.
Experiments are highly targeted, short duration, fast and low-cost. Each experiment is designed to test underlying product and market assumptions.
“If assumptions are left untested prior to the launch of a new product, service or experience, they present significant risk in the project delivery process”
Experimentation helps you to shift the odds more in your favour by testing and developing new initiatives in-market with customer prior to scaled launch.
Experimentation is all about being able to learn faster and make decisions faster.
If you can outlearn your competitors, you have a natural competitive advantage.
Experimentation example - AirBnB rejects social proof statements
In this experimentation example, AirBnB was seeking to understand if adding a form of social proof on their host signup landing page would encourage hosts to list their property.
The experiment consisted of a Control (Experiment A) without a social proof statement, and two Variants (Experiment B) and (Experiment C), containing different social proof statements.
Without engaging with the customers (hosts) that participated in this experiment to conduct additional research to understand user behaviours and decision-making drivers, it’s difficult to understand why Experiment B and Experiment C performed negatively without speculating.
For the purposes of the example, we’ll make some inferences:
Experiment A - Control
Experiment A is the control. There is no social proof statement.
Experiment B - Message 1 - Toronto is popular!
In Experiment B, this social proof statement suggests that, “39% of the listings in the local area of the host are fully booked next week”. This social proof statement is trying to influence hosts to list their property, given a level of quantifiable customer demand. The only problem is … 39% is not a very big number. For example, if 84% of listings in the local area were fully booked next week, it’s a much more compelling proposition for hosts, and maybe the experiment would have performed differently.
Experiment C - Message 2 - Host the world in Toronto!
For Experiment C, the social proof statement suggests that, “travellers from 180 countries searched for places to stay in your area last week”. Given that there are only 249 countries in the world, this social proof statement could give hosts the impression that the whole world is descending on their doorstep, raising a raft of other objections or fears?
The results
Experiment A (Control) was implemented. Experiment B and Experiment C were rejected after producing a negative outcome relative to the control experiment.
How do you conduct an experiment?
The process of experimentation involves the following key steps:
Developing a hypothesis
Designing an experiment
Executing the experiment
Analysing the results
Let’s discuss each of these steps in a little more detail.
Step 1 - Developing a hypothesis
A hypothesis states your predictions about what your research will find. It is a tentative answer to your research question that has not yet been tested.
For some research projects, you might have to write several hypotheses that address different aspects of your research question.
“A hypothesis is not just a guess — it should be based on existing theories and knowledge”
It has to be specific, testable and falsifiable, which means you can support or refute it through scientific research methods (experiments, observations and statistical analysis of data).
Your hypothesis must be able to be proven wrong.
Poor hypothesis:
Lightning is caused by angry ghosts.
Good hypothesis:
Lightning is caused by electrical charges moving from the earth to the clouds.
Don’t forget to set the key success metrics for your experiment before you launch, otherwise every experiment is a winner.
Step 2 - Designing an experiment
Good experimental design consists of an experimental group, and a control group to be able to compare the impact of changing a variable.
Experiments generally only change one variable at a time to isolate the impact. Otherwise, if you’re changing multiple variables simultaneously it can be difficult to pinpoint what’s caused any change.
When designing your experiment, you need to determine which variable you are going to change and the expected outcome from the experiment (I.e. increased conversion rate)
Split Testing (A/B Testing) is one of the most popular experimentation methodologies. For this method, you take two versions that are identical except for one variable (image, headline, discount) and compare their performance (click through rate, conversion rate).
Other factors to consider are the duration of the experiment, and your path to customer (E.g. Paid Advertising, EDM, Website/Landing Page etc.) and customer sample size.
Netflix has been iterating on showing additional fields upfront on their homepage. Previous experiments had demonstrated that displaying an email address field upfront increased customer conversion rates.
This experiment is the next step (Experiment B), to understand what impact adding in the password field upfront has on customer conversion rates.
Step 3 - Executing the experiment
The key to executing a good experiment is ensuring that variance is limited.
To have a valid experiment, the conditions between the experimental group and the control group need to be pretty much identical, except for the change in the tested variable.
Communication of your experiment is critical. You never want to announce to your target customers that your feature, product or offer is in fact an experiment.
That would immediately invalidate the experiment, and your results, as it would produce a change in consumer behaviour.
“It’s better to operate with intuition than incorrect data”
Experiment artefacts should always have the appearance of being real and available, without deliberately tricking customers.
Remember, you’re always experimenting with your customers, not on your customers.
Step 4 - Executing the experiment
Once the experiment has been concluded, it’s now time to compare the results of your experimental group against the results of the control group.
A degree in statistics is not required to do this.
You’re more looking to compare trends over time and the relative performance of different variables against one another.
It’s really important to try and analyse experimentation results as objectively as possible. We all bring our own beliefs, biases, opinions and assumptions to the table.
Be careful to not let decision-making be clouded by “group think”.
It’s very easy to read meaning into the data that isn’t there, or to create a counter narrative to the data that ensures an idea continues to be resuscitated, when it should be taken off life support.
The result of this experiment suggests a negative outcome, as Netflix reverted back to the Control version (Experiment A) - without the visible password field.
What are the benefits of experimentation?
Experimentation has the following key benefits:
Reduce business risk
Save costs
Decrease time
Better decisions
More options
Reduce business risk:
Experimentation decreases business risk by enabling business leaders and project teams to understand how an offer performs in-market at small-scale, prior to mass-market launch.
Experimentation provides strong indicators of customer demand and interest earlier in the development process. If experimentation results indicate strong signals of customer demand, it gives teams the confidence to keep incrementally investing.
Experimentation decreases the likelihood of False Negatives (you should’ve built it, but you didn’t) and False Positives (you build it, but you shouldn’t have).
Save costs:
Save capital by not developing new products or services that underperform in-market or provide little business impact.
Weeding out ideas that have no customer value or business impact early in the development process produces quantifiable CAPEX savings that can then be redistributed to pursue other business opportunities.
Decrease time:
Experimentation enables you to test and validate your product and market assumptions with real customers in terms of hours and days, not months and years.
As such, this allows businesses to accelerate to market faster, while ensuring that resources are prioritised to initiatives that have demonstrated business impact and customer value.
Better decisions:
Experimentation elevates the role of customer data and voice of customer in business decision-making, shifting the conversation from opinions and beliefs, to facts and data. This produces higher quality decisions, resulting in better business investments.
More options:
Good strategy is about creating options. Experimentation facilitates the generation of many options and possibilities, allowing business leaders to find the optimal path to reach their goals.
With experimentation being fast and low-cost, it enables businesses to pursue many more growth and expansion opportunities.
Who should be using experimentation?
Data-driven experimentation isn’t just for large, platform businesses like Amazon, Google and Netflix.
Experimentation is for all companies, whether you’re an early-stage startup, or a large, megacorp.
When is it best to use experimentation? From my perspective, all the time, at any opportunity.
Common applications of experimentation include (non exhaustive):
- New product development / growth opportunities
- Product line extensions / tactical offers
- New product features
- Impacts of pricing on customer demand
- New customer experiences
- Membership / Loyalty / Referral programs
- Customer retention and engagement
- Impacts of website design on customer conversion rates
- Impacts of Marketing campaigns / messaging on customer conversion rates
- etc. etc.
There is a myriad of opportunities. Let your mind run wild with possibility.
Conclusion
Organisations need to approach the development of new products and services differently.
Traditional big-bang models of project delivery carry significant risk. Product and market assumptions are only validated with customers when the new product is launched in the market.
Year on year, market failure statistics demonstrate that the majority of new consumer products will underperform in the market, failing to meet business expectations.
Experimentation helps business leaders to connect the dots between strategy, risk and business investment faster.
Good strategy is about creating options. Experimentation facilitates the generation of many options, allowing business leaders to find the optimal path to reach their goals.
Project teams can understand very early in the delivery process, which ideas deliver value for customers, and which don’t. Business resources can then be allocated to initiatives that show the most promise and deliver the greatest business impact.
Reduce business risk, save costs and decrease time with experimentation.
If you want to deliver more projects successfully, experimentation is the answer.
References:
Startup Genome, Wall Street Journal, Entrepreneur.com, Joe Newsum, Harvard Business Review, Harvard Business Review,
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.