My Proven Method For Writing An Experiment Hypothesis
A well-formed hypothesis is the foundation of a high-quality experiment. Your hypothesis is how you discover customer and business value. A hypothesis must be able to be proven wrong.
You may also enjoy these articles:
Introduction
A hypothesis is a prediction or theory about what your research may find. It is a tentative answer to a research question that is yet to be tested.
Hypothesis statements are formulated on existing knowledge or theories – they are not a guess.
A hypothesis must be specific, clear, testable and falsifiable – a hypothesis must be able to be proven wrong.
The problem is, if you aren’t being hypothesis driven, it’s very difficult to understand if your theories are supported or unsupported when you gather data.
In business, hypotheses are generally tested by a scientific process of experimentation. A hypothesis should be defined prior to commencing any experimentation process.
Are you on a road to nowhere?
Credit: Unsplash
Cue lyrics from the Talking Heads song, Road to Nowhere …
Well, we know where we're goin'
But we don't know where we've been
And we know what we're knowin'
But we can't say what we've seen
Imagine you set out on a roadtrip.
You packed the car, prepared the play list and headed off on your 1,000km journey. The only problem is, you don’t know where you’re headed.
You eventually arrive at your destination, however, it’s not what you were expecting.
You’ve been on a road to nowhere.
Running an experiment without a hypothesis is like starting a road trip just for the sake of driving, without thinking where you’re headed and why.
You’ll end up somewhere, but there’s a chance that you won’t have gained anything from the experience.
“Defining a hypothesis for your experiment is the most important step in the experimentation process”.
You shouldn’t be proceeding through your experimentation workflow until you’ve done the necessary critical thinking upfront to define a hypothesis.
I certainly don’t advocate running experiments without first defining a hypothesis.
Sure, you can run experiments without stating a hypothesis, but how will you know whether your efforts are successful, or not.
Setting hypothesis statements can be challenging in the beginning. It’s a skill that must be honed and refined over time.
In this article we’ll discuss six steps to creating a great hypothesis for your experiments:
Experiments and hypothesis
What is a hypothesis
Forming a hypothesis
How to write a hypothesis
Hypothesis checklist
Experiment like you’re wrong
1. Experiments and hypothesis
Experimentation is a form of scientific inquiry.
This process involves making observations and developing hypotheses. In product development, often, experiments are used to test the hypotheses.
Scientifically organised experiments must be carefully curated and designed.
“The hypothesis is the foundation of a high-quality experiment”.
Hypotheses are a critical step in the scientific method as they narrow down your line of inquiry, providing your research with more focus.
“A well-formed hypothesis is how you discover customer and business value”.
The reason that you’re running an experiment is that you’re looking to intervene in a system, making a change to something.
You’re looking to help people solve their problems and achieve their needs and desires.
This is achieved by changing and measuring user behaviour.
Affecting change often occurs through running an experiment to understand the impacts of executing a given change.
We predict that if we execute change X, then Y will occur. Are the impacts of the change positive, negative, or neutral?
“Every experiment provides a unique learning opportunity with your customers”.
Detailed observations of your customers and their behaviours should always support experimentation processes.
2. What is a hypothesis?
A hypothesis states your predictions about what your investigation or experiment will find. It is a tentative answer, that has not yet been tested.
“A hypothesis is not an idea”.
Ideas are a commodity. Everybody has them. They’re easy. Ideas are never proven or disproven. They’re just false facts.
There’s never any semblance of how an idea can potentially benefit a business or customer.
People who come up with ideas are Artists. People who come up with hypotheses think like scientists.
Simply, in the beginning, our hypotheses can be our working assumptions.
“A hypothesis is not just a guess – it is based on existing theories and knowledge”.
Your hypothesis should be informed by qualitative and quantitative research.
Importantly, your hypothesis has to be testable, which means you can support or refute it through scientific research methods (such as experiments, observations and statistical analysis of data).
“Your hypothesis must be able to be proven wrong”.
A strong hypothesis helps you to get the right data from your customers to help you advance closer to your business objectives.
No hypothesis, or a weak hypothesis, only helps to confirm your existing assumptions and beliefs, potentially taking you in the wrong direction, wasting precious business resources.
3. Forming a hypothesis
When conducting scientific experiments, your hypothesis should guide your experimental design.
Your hypothesis should be a suggested theory that is both testable and falsifiable.
I repeat, you must be able to be test your hypothesis, and it must be able to be proven wrong.
Credit: Quora
EXAMPLE A:
You have a hypothesis that “angry gods cause lightning”. You propose this as a possible explanation for why lightning occurs.
When the gods in the sky get angry, they unleash their fury, throwing barbs of lightning to the earth to punish humanity.
Is this a scientific hypothesis?
A. No. This statement is not testable or falsifiable
B. No. This statement is not testable
C. No. This statement is not falsifiable
D. Yes. This statement is testable and falsifiable
Answer: A
This statement is not a scientific hypothesis. The statement cannot be tested or proven wrong.
Credit: Unsplash
EXAMPLE B:
You have a hypothesis that “lightning is caused when electrons in storm clouds are attracted to protons in the ground causing an electrical discharge”.
Is this a scientific hypothesis?
A. No. This statement is not testable or falsifiable
B. No. This statement is not testable
C. No. This statement is not falsifiable
D. Yes. This statement is testable and falsifiable
Answer: D
This is a good example of a scientific hypothesis as the statement can be tested and proven wrong.
4. How to write a hypothesis for your product experiment
Forming a good hypothesis has five key elements:
1. The pre-work
2. The insight
3. The change
4. The impact
5. The metric
The statement follows “based on [qualitative/quantitative insight], we predict that [product change] will cause [impact] to [metric]”.
A proven method for writing an experiment hypothesis
Let’s look at each component in more detail and walk through an example.
1. The pre-work
Doing the necessary thinking and pre-work upfront will ensure that your research is focussed on solving the right customer problems, while having the biggest business impact.
Anchor to business strategy:
All the ideas and hypotheses that you test will need to be firmly anchored back to broader business objectives – business vision, strategies, KPI’s and goals.
The experiments that you conduct should be aimed at progressively moving the business closer towards these strategic objectives over time.
Ideas that don’t support your business advancing closer towards its strategic objectives should be strongly scrutinised and questioned.
What do you know?
What do you already know about the problem or opportunity that you’re trying to solve for?
There are many different qualitative and quantitative data sources that can be leveraged. Ensure that your data is objective, not subjective.
Take the time to do a thorough review of all the information that you’ve collected.
What are the key insights that have emerged from this discovery process?
Where should you be pointing your guns? Which problems should you tackle first?
Defining the opportunity:
You’ll need to choose a starting point to get going.
What is the business opportunity or customer problem that you’re going to investigate first?
There will be many opportunities that you could potentially investigate, however, you’re going to need to prioritise.
I think that it’s a good idea to always prioritise based on what’s going to have the biggest, macro-level business impact.
Some high-level criteria could include:
Biggest customer problem (I.e., High bill / customer bill shock)
Biggest performance impact (I.e., Sales uplift)
Biggest audience size (I.e., 000’s vs 000,000’s of customers impacted)
Start by testing your riskiest assumptions first – the things that if they were disproven would have your hypothesis proven wrong.
Focus on things that are going to improve customer experience and business performance.
2. The insight
Before you make a start, it’s important to know as much as possible about the problem that you’re trying to solve.
“Every hypothesis should be informed by some form of quantitative or qualitative research”.
What are the data, insights, information or facts that have led you to form your hypothesis?
“A good hypothesis is grounded in some form of reality. It’s not a guess”.
Often, insights will be generated by conducting customer interviews, customer surveys and data analysis.
Take the time to unearth your customer problems, pains, needs, and desires. What are the emotions that customers are experiencing? Context is key.
Work out your customer Jobs To Be Done.
Example:
From interviews with new car buyer customers, insights suggest that the value proposition on your website does not resonate with customers.
Customers have indicated that the ‘safety of their car’ is most important so that they can protect their family in the event of a crash.
You hypothesise that by changing the value proposition on your website from ‘price-led’ to ‘safety-led’ will increase the number of new car sales.
KEY TAKEAWAY:
Do your homework upfront. Your hypothesis is informed by insights that articulate the “why” behind your experiment and what you’re seeking to learn.
3. The change
This is your independent variable – the product change (variable) that you’re looking to manipulate during the experiment.
Generally, it’s best to try and isolate a variable for your experiment.
“Changing multiple variables simultaneously can make it difficult to pinpoint what’s caused a change in performance, for better or worse”.
Keep things simple - focus on one change at a time.
Will you be testing a micro change - an email subject header change, a different CTA, a new way of articulating the benefits of your product or changing the media from text to video?
Making small, micro changes will likely have lower impact. For example, changing the position of a button will likely yield a smaller improvement.
However, lots of little, micro-level step changes can manifest into a bigger impact over time.
Macro level changes (redesigning a home page) could produce an improvement of 100%+.
“Macro changes are much more complex, requiring a lot more time, resource, and budget. A strong organisational commitment is required to elicit macro changes”.
Every business decision has an opportunity-cost.
The product change that you select will be impacted by many variables – time, budget, resources, business strategy and organisational commitment.
However, as a rule of thumb, chase what’s going to have the biggest impact on business performance and customer experience.
Try and resist the urge to chase Local Maximums.
An experimentation team that is new, with a low-level of maturity is probably best placed to chase small, simple experiments to get runs on the board and develop positive proof points.
A more mature experimentation function can tackle bigger, strategic business opportunities, coming from a position of a highly trusted and respected business partner.
If you’re still finding your way with experimentation, don’t bite off more than you can chew.
Example:
Currently, on your company website, the value proposition copy above the first fold is framed from perspective of ‘price-led’. For your experiment, you’re going to change this text to articulate the value proposition using car ‘safety-led’ messaging.
KEY TAKEAWAY:
Think carefully about the variable that you’re going to change. If you’re running an A/B test isolate one discreet variable. Select one variable that you can change that will potentially impact your business objectives.
4. The impact
This is your dependent variable - the predicted outcome or change you’re observing during your experiment.
Your outcome should tie into the business objectives or KPI’s that you’re trying influence.
This could be more clicks, more sales or increase sign ups.
“It can take many months (or years) to potentially impact a macro business objective like growth or churn”.
It’s best to try and use a proxy metric that is indicative of the macro business objective that you’re trying to influence.
Example:
The broader business objective that you’re trying to influence is new car sales. A good proxy metric to measure short-term impact could be increasing sign ups to car demos.
KEY TAKEAWAY
Think about the desired outcome of your experiment. Establish a relevant baseline so that you can quantify change. Choose a short-term proxy metric as a leading indicator to broader, strategic business objectives if required.
5. The metric
Use existing research and data to understand current business performance so that you can establish an informed baseline.
What is the baseline metric that you’ll measure? Think carefully to ensure that you’re gathering data to measure the right outcome.
Keep it simple – don’t try and measure too many metrics at once.
“The metric that you set needs to realistic. It should be informed by prior information and learnings”.
What is the change that you expect to see relative to the existing baseline? Is it a smaller, incremental change, or a larger macro change?
You shouldn’t be making a wild guess. If you don’t have existing data or information available internally, look to industry guidelines or benchmarks to get you started.
You can always refine your success metric after running initial experiments.
Also, think about the magnitude of the change that you expect to see.
For a low-complexity, fast, low-cost implementation you may be happy with a smaller, incremental performance improvement.
If the change is high-complexity and high-cost, the magnitude of the performance improvement will need to be substantial enough to warrant the required business resource commitment.
Example:
You predict that you can increase new car demos by 17% through implementation of your product change.
KEY TAKEAWAY
Conduct data analysis and research to set an informed and realistic success metric. It’s always helpful to understand baselines for existing experiences. Make sure that you’re gathering the right data to evaluate your success metric.
Putting it all together
Based on [insight] 43% of US Volvo new car buyers indicating that car safety is their number one purchasing driver, we predict that [change] repositioning the website value proposition header from ‘price led’ to ‘safety led’ will [impact] increase the number of new car demos [metric] by 17%.
5. Checklist for a strong hypothesis
Below is a set of criteria to help you evaluate the strength of your hypothesis:
1. Is it related to my learning objectives?
- Is my hypothesis related to my learning objectives or is it completely different?
- How many hypotheses do I need to address my learning objectives? One or many?
2. Is it falsifiable?
- Does my hypothesis define measurable variables?
- Is there a method to analyse your data?
- Are you able to collect the right data to measure your hypothesis?
3. Is it neutral?
- Is my hypothesis loaded? Have I included individual biases?
- Am I testing this hypothesis just to prove somebody wrong?
4. Is it simple?
- Can we easily collect data to test the hypothesis?
- Is there any specialised know-how required to gather data to test the hypothesis?
5. Is it specific?
- Did I define my variables clearly?
- Is the hypothesis unique to the opportunity space that I’m investigating?
6. Experiment like you’re wrong
This is where the rubber hits the road. Don’t bring all your good work unstuck with poor execution of your experiment.
The goal of your experiments should never be to prove your hypothesis right. It’s not possible to prove a hypothesis to be true.
“Run your experiments like you’re wrong”.
Your hypothesis is either supported or unsupported by the data and evidence that you collect.
Assume that your change will have no effect (null hypothesis).
If you can measure a statistically significant change in your key metric, then you can reject the null hypothesis and conclude that there is a positive effect.
You’ve likely discovered a change to your system that will improve business performance and customer experience.
“Discovering a ‘winning’ experiment should be the exception, not the norm”.
Very few of our ideas are actually good ideas. The failure rate of your experiments should be high.
If you’re consistently discovering a high number of winning experiments, it’s more than likely down to poor experiment design and analytical errors.
Conclusion
Everything starts and ends with your hypothesis.
A well-formed hypothesis is the foundation of a high-quality experiment. Our hypothesis is how we discover customer and business value.
Setting testable and falsifiable hypothesis statements is the foundation of a disciplined process of scientific experimentation.
Preparing clear and concise hypothesis statements is a skill best practised over time. The more you practise, the more natural this process will become.
A hypothesis is never right or wrong. It is either supported or rejected by the evidence and data that you collect.
Expect that most of your hypotheses will be disproven.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.