The Experimentation Tax – How to Avoid Paying This Hidden Cost of Experimentation

---

“There is an Experimentation Tax associated with all experimentation programs – new and old, large and small. The Experimentation Tax is never zero. When the Experimentation Tax is high, it acts as a deterrent, decreasing motivation, interest, and excitement of teams to perform experiments”.

---

You might also enjoy these articles:

• Lukas Vermeer - A Global Leader in Online Experimentation

• Strengthen Your Strategy With Experimentation

• Be Hypothesis Driven, Not Idea Led

• Six Lessons From The Award Winning bol.com Experimentation Program

• Create Communities of Practice to Scale Experimentation Faster

• Yelp: A Story Of Building An Experimentation Program

Introduction

No one likes paying taxes. Me included.

We’re all looking for creative ways to reduce the amount of tax we pay, especially personal income tax.

The Experimentation Tax should be viewed no differently.  

We should be constantly on the lookout for improvement opportunities to reduce the human cost of experimentation in our experimentation flywheel.

Oftentimes, we can completely overlook the human efforts associated with running our experimentation processes, neglecting the friction and frustrations imparted on users.

We can spend a lot of time doing things that aren’t value-added.

If performing experiments is too complex and time consuming, business teams will opt out, choosing the path of least resistance – not testing new opportunities before release.

Exactly what we don’t want.

If we put our Product Designer hat on, we would never in a million years subject our customers to the same level of friction that we knowingly tolerate in our experimentation processes, day in, day out.

Instead, we should apply the same sort of rigour and high standards that we apply to developing customer experiences, to the development of our experimentation platform and processes.

Interactions with experimentation processes should be easy, effortless, and simple.

In this article we discuss:

1. What is the Experimentation Tax?

2. Why is the Experimentation Tax important?

3. A guide to measuring the Experimentation Tax

4. How to decrease your Experimentation Tax
   
   
   
   

1. What is the Experimentation Tax?

The Experimentation Tax is the “human cost” associated with designing, executing, monitoring, analysing, and reporting on an experiment.

2. Why is the Experimentation Tax important?

Complex, superfluous processes and manual handling act to increase the cost of the Experimentation Tax.

When the Experimentation Tax is high, it acts as a deterrent, decreasing motivation, interest, and excitement of teams to perform experiments.

This is an expensive price to pay for any organisation.

Lowering the human cost of experimentation does not get enough attention in most experimentation programs.

Commonly, it is one of the most neglected areas.

“Continuously lowering the cost of A/B testing is a critical step in the flywheel that is often missed. If running A/B tests remains costly, A/B testing will remain limited to highly interested early adopters with a lot of resources.” - Fabijan et. al. (2021)

Over time, you should be looking for ways (solutions and continuous improvement opportunities) to consistently reduce the human cost of experimentation, year on year.

There is an Experimentation Tax associated with all experimentation programs – new and old, large and small.

The Experimentation Tax is never zero.

However, in your organisation, you need to ensure that the Experimentation Tax rate that you’re currently paying, is as low as possible.

To decrease your Experimentation Tax rate, it requires an ongoing investment of capital, time, and resource to streamline your experimentation system.

3. A guide to measuring the Experimentation Tax

Key questions to answer

If you want to scale your experimentation culture and program, you need to identify where the friction points are in your experimentation flywheel.

Two important questions to answer:

1. What’s currently limiting you from performing more experiments?

2. Where are the bottlenecks in the experimentation process?

In his book The Goal, Eliyahu Goldratt introduces the Theory of Constraints, a methodology for improving operational efficiency and performance.

The key premise of the book is that every system has at least one constraint (or bottleneck) that limits its overall performance.

Identifying and removing the bottlenecks in your experimentation program value chain is critical for improvement.

Constraint detection

It’s critical to identify and understand the constraints / bottlenecks in your experimentation value chain.

“The unexamined life is not worth living” - Socrates

Time and Motion Analysis:

If you don’t have experimentation program metrics in place to monitor testing efficiency, you will need to conduct a series of Time and Motion analyses to baseline cycle times for key testing phases through your experimentation system.

The objective of your Time and Motion analysis is to understand how long it takes to perform a standard experiment, at a normal pace, under normal conditions.

A Time and Motion analysis looks something like this:

1. Analyse what you’re doing, how you’re doing it, and how long it takes

2. Identify improvement opportunities

3. Make a change to how you work

4. Measure the effectiveness of the change

5. Rinse and repeat

Remember, lots of small changes are compounding and add up.

Testing Efficiency metrics:

Good experimentation programs will track Testing Efficiency to understand how well they are using “resources” to perform experiments.

These metrics can be helpful for identifying continuous improvement opportunities to improve experimentation process efficiencies.

Some common metrics include:

• Experiment cycle time: the average time taken (days/weeks) end-to-end to perform an experiment, including design, execution, run time, analysis, completion

• Experiment configuration time: the average time taken (hours/days/weeks) to design and execute a prioritised experiment (does not include experiment run time)

• Experiment run-time: measures the average amount of run-time (days/weeks) it takes for an experiment to complete

• Results to insight: the amount of time (hours/days) it takes to analyse and compute experiment results / outcomes and produce the experimentation scorecard

• Cost per experiment: measures the total business cost of performing an experiment, including design, build, execution, and analysis (hours / days / $$)

Theory of Constraints:

Eliyahu Goldratt outlines Six Focussing Steps for Theory of Constraints:

1. Analyse the system: how is work moving through the experimentation system

2. Identify the constraint: find the process that limits the system output

3. Exploit the constraint: maximise efficiency of the bottleneck

4. Subordinate everything else: align other processes to support the constraint

5. Elevate the constraint: invest in increasing capacity of the bottleneck

6. Repeat: once a constraint is broken, go back to Step 1

Think of your experimentation system like a chain. The idiom of Theory of Constraints is “the chain is not stronger than the weakest link”.

The weakest link is an area of vulnerability for your experimentation system, and the organisation.

Once you’ve fixed the weakest link, there will be another link that breaks, and requires attention. This process is perpetual.

Strengthen the weakest link in your experimentation system applying Theory of Constraints

Tips:

• Your work is never done – there’s always more work to be done to improve experimentation systems. It’s a destination you never reach

• Spot backlogs – it is common for Work in Progress (WIP) to pile up before a constraint. Observe where excess “inventory” is piling up

• Focus on system flow, not resources – aim to improve the overall experimentation system flow, rather than the efficiency of individual resources (Global Maxima versus Local Maxima)

• Aim for increased throughput – optimise the system to increase the rate at which experiments can be processed through your workflow processes (I.e., more inventory/experiments processed through your machine)

• Role of leadership – communicate clearly about your system constraints, goals you’re working towards, and investment/resources required

• System measurement – use relevant and meaningful metrics to monitor and track the performance of your experimentation system

4. How to decrease your Experimentation Tax

There are many ways that an organisation can decrease the amount of Experimentation Tax they’re paying.

All businesses and experimentation programs are unique, characterised by their own individual challenges. Every experimentation program is different from the next.

However, there are many common tasks that exist across all programs that can be simplified or automated.

Below are some universal focus areas where you can look to start paying down your Experimentation Tax.

This list is not intended to be exhaustive, just a starting point.

Over time, big changes and small changes are compounding, all adding up to reduce the human cost (tax) of experimentation.

FOCUS AREA	SOLUTION
Terminology	Experimentation is a language game. Standardise experimentation language and terminology so that teams can have discussions with a mutual, shared understanding, creating efficiencies and reducing wasted time.
Onboarding	Provide onboarding support to nascent teams who are commencing their journey with experimentation. Help teams to quickly understand experimentation systems and processes so that they can start deriving value from experimentation as fast as possible – while motivation and interest is still high.
Documentation	Develop process documentation, user guides and checklists so that new teams / new starters can make a fast start with experimentation. Teams shouldn’t have to embark on “choose your own adventure”. Good quality documentation enables teams to solve problems through self-service.
Education & support	Conduct ongoing education and support programs with business teams to build capabilities in core experimentation skills and methods etc. Aim to decrease the reliance of teams on expert support to perform experiments.
Processes	Conduct periodic review of experimentation processes, identifying friction points and bottlenecks. Formulate a portfolio of continuous improvement opportunities. Aim to tackle blockers that are a rate limiter to experimentation throughput.
Champions	Implement a network of cross-functional, domain experts who can support on the ground with experiment design, experiment execution and issues resolution. This enables teams to run faster, performing higher quality experiments.
Templates	Develop standardised templates for key experimentation artefacts to ensure consistent, transparent, high-quality documentation. Data and information is accessible across the organisation, captured centrally and searchable.
Planning	Statistical planning is conducted with simple, user-friendly tools. Teams can easily provide data inputs and receive sound planning outputs (E.g., Sample size) in return.
Design	Make it easier for teams to implement and setup experiments by simplifying coding requirements.
Launch	Safe rollout and ramp up of experiment launch can be automated.
Monitoring	Implement automated alerts for issues detection, data quality checking, breaches of guardrail metrics etc. Experiments can be auto stopped.
Reporting	Results analysis and data analysis is automated and summarised in a UI. Interpretation of experiment results should be intuitive and user-friendly to avoid misinterpretation. Teams can self-service on data analysis if they want to deep dive.
Communications	Develop system integrations with communication platforms (I.e. Slack) to distribute experimentation results, alerts, comms, and insights at scale automatically.
Knowledge Management	Ensure that all experiment documents, outcomes, and results are stored in a centralised, searchable knowledge management system. This avoids lots of wasted time and effort connecting multiple information sources across disparate systems to establish a single source of truth.
Program automation	This is a big one, requiring investment, time, and resource over many years. It will likely encompass many of the above areas. Aim to automate common processes, repeatable tasks, and low-level manual handling. Identify automation opportunities in your experimentation lifecycle, prioritise high-impact areas and develop a roadmap of automation opportunities. For example, automate computation of statistical analysis and data analysis.
Infrastructure	Centralisation of experimentation infrastructure into one connected system provides a solid foundation for automation efforts. While requiring the biggest investment (time, money, resource), investing in infrastructure provides outsized gains for simplification and throughput improvement. Operating an experimentation ecosystem across multiple disconnected systems makes automation much more difficult.
Metrics	Make it as easy as possible for teams to understand experimentation metrics, add new metrics to their experiments, and integrate metrics to their analyses and scorecards.

In summary

The Experimentation Tax is the “human cost” associated with designing, executing, monitoring, analysing, and reporting on an experiment.

Every experimentation program is paying some form of Experimentation Tax, some more than others. The Experimentation Tax is never zero.

Aim to decrease the amount of Experimentation Tax that your organisation pays!

Some things to consider:

• Every experimentation system has at least one major constraint

• Analyse and identify blockers/constraints in your experimentation value chain

• Streamline waste, complex processes and manual handling

• Develop an automation roadmap for your program

• Apply a Product Designer mindset – don’t tolerate poor user experience

Remember, the work of improving an experimentation program is never done.

It’s a destination you never reach.

---

Need help with your next experiment?

Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.

Are you struggling with experimentation in any way?

Let’s talk, and I’ll help you.

---

References:

Experimentation Masters Podcast