Sequence Experiments to Decrease Risk and Uncertainty
Many experiments are required to test a business hypothesis. Sequence experiments to increase evidence strength from customers so that you can reduce risk and uncertainty.
You may also enjoy these articles:
Introduction
Experimentation aims to decrease uncertainty and risk over time, providing business leaders with more confidence that they’re on the right track.
Experimentation should be viewed as a journey, rather than a one-off event.
The journey starts with fast, low-cost experimentation techniques with lower evidence strength. As business confidence in an idea increases over time, the complexity, cost and sophistication of experiments will also increase in order to gather higher-quality customer data and evidence.
The results from one experiment should never define a strategy or a project. It’s dangerous to make important decisions off the back of one experiment with weak evidence.
Project teams should look to sequence experiments together to try and build a stronger evidence base from customers over time.
Several experiments are always required to validate or invalidate a hypothesis.
What is research evidence strength?
All forms of customer research are not created equally.
Customer research methods invariably hold varying degrees of research evidence strength, some stronger than others.
Research methods that ask customers for their opinion and feedback on intentions or future behaviours will score lower on evidence strength than research methods that measure actual customer actions and behaviours.
Qualitative research methods provide greater depth and breadth of customer insight, being really helpful to generate new ideas and understand customer intent, pain and problems. This type of research is great for helping to answer the “why” something happened.
Quantitative research methods such as experimentation, provide a narrower and shallower research perspective, however, are an excellent way to measure “what” customers do – their actual behaviours and actions when presented with an offer or proposition.
As we can see, both qualitative and quantitative research practices have pro’s and con’s.
No one research method is superior to the other.
There’s upside and downside with all methods.
This is why different research methods are best used in concert, to provide a more holistic, 360-degree view of the customer and problem to be solved.
Otherwise, you’re going to create organisational blind spots if you have a dependency or over-reliance on just one research method. You’re going to miss understanding the “why” or the “what”.
Qualitative insights inform directionality and framing of your product construct by understanding customer jobs, pains and problems in more detail.
Quantitative data from experiments should always be the ultimate gatekeeper to product decisions and business investments.
Product decisions and business investments should never be made on qualitative insights alone.
Why is experiment sequencing important?
Sequencing experiments is important as it enables project teams to generate a stronger evidence base from their customers over time.
This allows you to reduce risk and uncertainty as much as possible before you’ve built your solution.
Experimentation decreases risk & uncertainty over time
You don’t need to build anything to test your business hypothesis.
The bigger the business investment to build the solution, the more you need to run multiple experiments to validate customer jobs, pains and gains that you assume they have - your assumptions.
This approach has the following key benefits:
Greater business confidence
Reduces risk
Less doubt and uncertainty
Stronger indicators of customer demand / interest
More successful business investments (ROI)
All business strategies are built on a set of working hypotheses.
In order to most effectively test your hypothesis, it’s important to run a series of experiments to validate or invalidate your hypothesis.
It’s always really dangerous to run one isolated experiment and make an important strategic business decision from an incomplete investigation, or on weak or low customer evidence.
If your hypothesis is proven wrong then you need to change your strategy.
If there’s no negative signals and, your hypothesis is validated, continue progressing forward.
How to sequence your experiments?
The key objective of experiment sequencing is to progressively increase the level of customer commitment over time.
The more confidence you have in your idea, you should look to keep increasing the “skin in the game” from your customers.
This process is how you move from a position of low confidence and low evidence strength, to a position of higher confidence and high evidence strength.
A good way to think about testing an idea is like a narrative, or story.
A good story has the following attributes:
- A beginning
- A middle
- An end
The beginning
At this point, you really don’t know anything about the quality of your idea.
You want to gather evidence and data as soon as possible to commence testing your hypothesis.
Start the process by gathering fast and low-cost forms of evidence.
This could include the following:
- Review existing business performance data
- Review feedback from customer calls
- Research on forums/discussion boards
- Jack into customer calls with the Customer Service team
- Analysis of search trends/data
- Conduct customer interviews
- Conduct customer surveys
- Conduct industry research
- Feedback from Sales/Marketing teams
What you’re after here is directionality. Are you headed in the right direction or not?
Is my working theory true or false?
You can afford to start out with lower forms of customer evidence strength as, if you’re still on the right path, you can generate stronger evidence later.
The middle
From your preliminary data gathering exercise, all indicators suggest that you’ve identified a customer problem to solve.
Now, we want to start amplifying the strength and quality of the data that you’re getting from your customers.
This will involve running disciplined and targeted experiments to test customer demand and interest in your proposition.
You’re also starting to get sharper on value drivers so that you can better understand how customers interact and behave with your new offering, what they value and what’s important to them.
Some experiments that you could run include:
- AdWords campaign driving traffic to a landing page
- Email campaign
- Explainer video
- Brochure with QR code driving traffic to a landing page
- Social media campaign
- In application forms/modals
- In application customer tasks or actions
- New feature in existing product (I.e. Ask an expert)
- A/B testing
During this phase of the story, we’ve ramped up the evidence strength from Low to Moderate levels.
As you move through The Middle phase, it’s important to progressively increase the level of “Skin in the Game” from your customers.
You want to ensure that the demand indicators from customers is still strong enough to warrant an ongoing investment in time, resource and business capital.
At this stage, if customer interest starts to waver, declines significantly or there’s zero interest, once you’ve asked for increasing levels of customer commitment, it’s a warning sign and you need to sit up and take notice.
If the customer is unwilling to perform a click, take an action or provide some form of personal information, the likelihood that they’ll pay for your product is low.
The end:
If you’ve reached the end of the story, you’re still seeing positive demand indicators and customer interest. This is a good sign.
Customers are interested in your offer. Your solution solves a problem for the customer.
At this stage, you’re looking to use experiments that provide the strongest evidence possible.
These types of experiments are typically the most sophisticated, costly and take the longest time frame to execute.
The strongest form of evidence from a customer is always a commitment to pay.
If possible, you want to try and capture a customer payment and deliver the offer. This will be to a small subset or targeted cohort of customers. Remember, we’re still in learning mode.
Be content for your customer experience to not be pixel perfect. However, if the customer experience is awful, and detrimental to overall user experience, it’s going to impact experimentation outcomes, potentially providing a false negative.
If you’re unable to capture a payment from the customer, try taking your experiment/s as close as possible to capturing a payment, to measure a firm customer action or behaviour, without actually taking a payment.
Types of experiments that you could potentially run during this phase include:
- High touch Concierge style onboarding
- Product pre-sale
- Signed Contract / Letter of Intent
- Upfront payment / Subscription payment
- Wizard of Oz (manual execution of future automated tasks)
- Mechanical Turk (“high fidelity” UI with manual backend)
To reiterate, reduce risk and uncertainty as much as possible before you build anything.
Examples of experimentation sequences
A few key points for your experimentation sequences:
Start off fast and low-cost so that you can commence gathering evidence and data to test your hypothesis as early as possible
Progressively increase the evidence strength of your experiments over time. Start with low evidence strength, building up to higher evidence strength if customer demand indicators are still strong
Select experiments that are fit for purpose based on time, cost and resource constraints
Be mindful of balancing business risk appetite with the types of experiments you run. If your business is more risk averse, don’t select experiments that are perceived to be higher risk
All experiments sequences are not the same. Experiment sequences will differ based on the type of business model – B2C, B2B and B2B2C
Two common examples include:
Business to Customer (B2C):
B2C experimentation is the most common and simplest form of experimentation where a business sells a product or service directly to a consumer. Examples include Amazon, Target, Netflix and Google. Some of these companies are B2B too.
These companies have digital platforms and multiple channels that enable many different types of experiments to be executed quickly at low-cost.
Example B2C experimentation sequence:
Business to Business (B2B):
Traditional enterprise B2B relationships (SAP, Oracle, IBM, Adobe) can be a little trickier for experimentation due to the high value of the business relationship. While it’s not impossible to run B2B experiments, you just need to be mindful of balancing risk and reward.
SaaS B2B providers (Atlassian, Hub Spot, Survey Monkey, Shopify) all actively engage with their customers and user base to develop new solutions to emerging customer needs. Many B2B SaaS software buyers are now looking to try before they buy. There are many experimentation opportunities in this space – new features, new product offerings, retention, upgrading, customer onboarding etc.
Example B2B enterprise experimentation sequence:
Conclusion
Experimentation is a journey over time, not a one-off event.
The evidence strength of your experiments should increase over time, in line with growing business confidence.
Aim for fast and low-cost forms of evidence at the beginning of your journey as you commence validating your hypothesis.
Increase the sophistication and complexity of experiments as you learn more and customer demand indicators for your new offering remain strong.
Customer research methods are not created equally. Combine qualitative and quantitative research to provide a three-dimensional view of the customer and problem you’re trying to solve.
This will reduce the chance of creating gaps in your evidence base and introducing organisational blind spots.
Think about creating a story or narrative with your experiments – a beginning, middle and end.
By the end of your experimentation journey you should’ve reduced risk and uncertainty to the point that you’re confident to go ahead a build your solution.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.