Experimentation Decision-Making: How To Improve The Quality Of Your Decisions

---

“The raw data from experimentation results is objective. It’s not someone’s opinion about what happened. The data was collected objectively. However, every time we make a jump from concrete data to sense-making, to develop meaning and draw conclusions, we weave in layers of abstraction. The further we jump from real, concrete data, the more abstract our understanding of the data. What we can be left with in the end may only represent a figment of our imagination”.

---

You might also enjoy these articles:

• Lukas Vermeer - A Global Leader in Online Experimentation

• Strengthen Your Strategy With Experimentation

• Be Hypothesis Driven, Not Idea Led

• Six Lessons From The Award Winning bol.com Experimentation Program

• Create Communities of Practice to Scale Experimentation Faster

• Yelp: A Story Of Building An Experimentation Program

Introduction

All product teams should be experimenting. If you’re not experimenting, you’re guessing.
 
Product teams should be using experimentation to inform their product development decisions. It should be an embedded step in the product development process or release cycle.
 
Experimentation is a mission critical strategy for high-quality decisions and risk mitigation.
 
Experimentation provides decision support to our product intuition. It helps to keep a tight rein on what we think customer want and determining what they actually value.

All good Product Managers have strong product intuition. Possessing sound product intuition is not bad, it comes naturally with developing a deep understanding of a particular domain.
 
If there’s a gap in this understanding it represents a blind spot. Organisational blind spots can produce sub-optimal decisions or decision-making errors.
 
Blind spots can result in the waste of precious business resources – time, money, and people.

There’s never been a point in history where we’ve had so much data on our customers, yet we still know so little. There is still a massive disconnect between what we think our customers need, and what they truly value.

In this article we discuss two things:

1. Why establishing meaning can be challenging

2. How to improve your decision quality
   
   
   
   
   

1. Why establishing meaning can be challenging

The experimentation process is not objective

Experimentation is often positioned as a fool-proof, objective decision-making tool.
 
Only, it’s not.
 
While experimentation results and data are objective, the experimentation process is still subjective.
 
Experiment prioritisation, experiment design, results interpretation etc. can be somewhat subjective.
 
On experimentation results - the raw data from experimentation results is objective. It’s not someone’s opinion about what happened. The data was collected objectively.

Challenges with information processing

Things can start to unravel when we interpret experimentation data.
 
Experimentation results can be open to interpretation, just like any other data.
 
Every time we make a jump from concrete data to sense-making, (I.e., establish patterns and themes), to develop meaning and draw conclusions, we weave in layers of abstraction.
 
We often don’t even realise it, but we’re inherently overlaying our own biases through the data analysis.
 
The further we jump from real, concrete data, the more abstract our understanding of the data.
 
What we’re left with in the end may only represent a figment of our imagination.

We’ve moved from something that was real to something completely different.

This is hard to avoid. It’s how humans perceive and make sense of information.

For instance, where there is an absence of information, we fill in the blanks with our own narratives, stories, and theories to create patterns and meaning.

Also, we often simplify the meaning in the data to make things easier for ourselves.

To overcome these challenges, there’s some practical measures we can use to improve our decision quality.

Establishing meaning in data - Moving from concrete to abstraction

2. How to improve your decision quality

To help improve your decision quality, there’s four practical steps that you can take to level up your game:

1. Document decisions

2. Socialise decisions

3. Ask questions to dig deeper

4. Evaluating experimentation results

1. Document decisions

Think about establishing a centralised repository to store key product decisions.
 
This information should be freely available to people in the organisation if anyone wants to revisit past decisions.

• What was the decision?

• What was the outcome?

• When was the decision made?

• Who made the decision?

It’s important to store key decisions so that decision performance / quality can be analysed and reviewed over time.

Did the decision have the impact we thought it would have on business performance or customer experience?

Where decisions produce negative impacts to the customer or business, they represent organisational learning opportunities.

2. Socialise decisions

Reviewing and interpreting experimentation results shouldn’t occur in a vacuum. Product Managers aren’t a lone wolf making decisions in isolation.

When it’s time to take the next step, bring the team together and socialise the decision-making process.

While this isn’t an attempt at decisions by committee, it’s an opportunity for cross-functional team members to provide input and feedback.

The people who best understand how experimentation works need a platform to share different perspectives and contribute to product decisions.

3. Ask questions to dig deeper

When the time is right, and you’re ready to decide, go deep by asking questions to challenge beliefs and assumptions.

This approach will help you to move towards a higher quality decision.
 
Some questions to consider:

• What was our hypothesis?

• What experiments did we perform?

• What did we think was going to happen?

• What actually happened?

• What do we think that means?

• Who has a different perspective? If so, what do you think it means?

• What decision are we going to make?

• Why is the decision the correct one?

• When was the decision made?

• Who made the decision?

4. Evaluating experimentation results

Once your experiment is complete, you want to harvest as much insight from the learning experience as possible.

Specifically, you want to be able to understand how you need to act - implement the solution, iterate the solution or stop.

Use a framework or model to help evaluate experimentation results.

The below framework is a good tool to help you think more objectively about experimentation results, decisions and next actions.

To recap

Good Product Managers develop a strong product intuition. Inherently, they have a good sense on product directionality and customer motivations and needs.

Objective data from customer experiments provides decision support to our product intuition, helping to remove blind spots.

Some considerations to improve your product decision quality: 

1. Support your product intuition with objective experimentation data

2. Understand how layering abstraction with analysis gets you further from the truth

3. Document decisions to improve long-term decision-making performance

4. Socialise product decisions – give cross-functional teams a voice

5. Challenge beliefs and assumptions by going deep with questions

 
Be aware of the challenges and limitations with human information processing systems.

Make sure that you’re not introducing biases and unnecessary abstractions when analysing and interpreting results.

---

Need help with your next experiment?

Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.

Are you struggling with experimentation in any way?

Let’s talk, and I’ll help you.

---