A Comprehensive Guide To Experimentation Program Metrics
“Experimentation program metrics help you track the progress of your experimentation program, identify areas for improvement, and make informed decisions about how to allocate your resources”.
You might also enjoy these articles:
Six Lessons From The Award Winning bol.com Experimentation Program
United Nations: Solving The World’s Toughest Challenges With Experimentation
Introduction
Experimentation has become a cornerstone of modern business strategies. Companies use experimentation programs to optimize and innovate various aspects of their operations, from user experiences, product performance, marketing campaigns or algorithms.
Experimentation program metrics play an important role in assessing the performance and health of an experimentation program.
These metrics can help you track progress of your experimentation program, identify areas for improvement, and make informed decisions about how to allocate your resources.
In this blog post, we will delve into the significance of experimentation program metrics and explore the different types of metrics that can help you measure the effectiveness of your experimentation program.
In this article we discuss:
Why experimentation program metrics are important
Types of experimentation program metrics
A guide to experimentation program metrics
1. Why experimentation program metrics are important
Experimentation program metrics provide quantifiable data and insights into the performance of your experimentation program.
“Without monitoring the ongoing performance of your experimentation program, it's challenging to determine the success and impact of the program, making it difficult to refine experimentation strategies effectively”
There are several reasons why experimentation program metrics are important:
To track progress: Metrics can help you see how your experimentation program is performing over time. This can help you identify areas where you are making progress and areas where you need to improve.
To identify areas for improvement: Metrics can help you identify specific areas of your experimentation program that need improvement. For example, if you are seeing a lot of bugs and issues with experiments, you may need to improve your experiment design and execution processes.
To make informed decisions: Metrics can help you make informed decisions about how to allocate your resources. For example, if you are seeing a high number of prioritised experiments piling up in your experimentation backlog, you may need to consider sourcing more Engineering resources, or commit resource to workflow automation to increase throughput.
2. Types of experimentation program metrics
Experimentation program metrics can be categorized into four main types:
Test Velocity
Test Efficiency
Test Quality
Test Effectiveness
Each type of metric offers valuable insights into different aspects of the experimentation process.
Test velocity: Measures how quickly you can perform experiments
Test efficiency: Measures how well you are using resources to perform experiments
Test quality: Measures the quality of your experiment design and execution
Test effectiveness: Measures the impact of your experiments on your business goals
3. A guide to experimentation program metrics
Test velocity metrics
Test velocity metrics measure the speed and frequency at which experiments can be performed. These metrics can be helpful for tracking your progress and identifying areas where you can improve.
Test Velocity metrics are a good indicator of how quickly your experimentation program is learning and iterating with customers.
Some common Test Velocity metrics include:
Ideas generated: the number of new opportunities/ideas generated within a specific timeframe (I.e., per month)
Experiment backlog: the number of prioritised experiments in your backlog waiting to be designed and executed
Experiment velocity: the total number of experiments that you have performed per month/quarter/year
Experiment cycle time: the average time taken (days/weeks) end-to-end to perform an experiment, including design, execution, run time, analysis, completion
Experimentation owners: counts the total number of experimentation owners that have performed at least one experiment
Test efficiency metrics
Test Efficiency metrics measure how well you are using your resources to perform experiments. These metrics can be helpful for identifying areas where you can improve your experimentation process efficiency.
Some common Test Efficiency metrics include:
Experiment configuration time: the average time taken (hours/days/weeks) to design and execute a prioritised experiment (does not include experiment run time)
Cost per experiment: measures the total business cost of performing an experiment, including design, build, execution and analysis
Experiment run-time: measures the average amount of run-time (days/weeks) it takes for an experiment to complete
Time to productionise: measures the average time taken (days/weeks) to launch a successful experiment
Scorecard compute time: measures the time taken to compute the experimentation scorecard
Expert support: measures the number of experiments where expert help was not required
Program automation: measures the number of user tasks that are automated in the end-to-end experimentation lifecycle
Test quality metrics
Test Quality metrics measure the quality of your experiment design and execution. These metrics gauge the reliability and robustness of your experiments.
These metrics can be helpful for identifying areas you can increase the reliability, quality and trustworthiness of your experiments.
Some common Test Quality metrics include:
Experiment error rate: measures the number of experiments where bugs/issues were identified post-launch
Statistical significance rate: measures the number of experiments that obtained a statistically significant outcome
Decisions affected: measures the number of business decisions positively impacted by experimentation
Experiment success rate: measures the total number / percentage of experiments that were implemented or productionised
Validated feature releases: counts the number of new feature releases that were validated through experimentation
Features not shipped: counts the number of features not shipped due to experimentation results
Valid experiments: counts the number of valid experiments where a trustworthy result was returned
Test effectiveness metrics
Test Effectiveness metrics measure and assess the impact of your experiments on KPI’s or organisational goals. These metrics assess how well your experiments contribute to business goals.
Some common Test Effectiveness metrics include:
Impact on OEC: measures the total impact of all experiments performed on your OEC over a period (I.e., per quarter)
Return on investment (ROI): measures the financial return on your investment in experimentation
Customer satisfaction: This metric measures the impact of your experiments on customer satisfaction
Experiment champions: counts the number of experimentation champions in the organisation
Experimenters trained: counts the number of people / teams that have attended formal experimentation training programs
In summary
Experimentation program metrics are fundamental for understanding the success and impact of experimentation efforts within a business.
Experimentation program metrics drive data-driven decision-making, more effective resource optimisation, and experimentation program continuous improvement.
By employing a variety of different program metrics, businesses can fine-tune their experimentation programs, ensuring they are on the path to success.
Through careful analysis of experimentation program metrics, organisations can optimise their experimentation processes, removing blockers and bottlenecks where necessary, to achieve greater scale and growth.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.