How British Airways Use Experimentation To Transform Customer Experience

---

“ If your experimentation program slows down the product release cycle too much, you need to improve experimentation processes to make the flywheel go faster. There’s no such thing as a finished experimentation program. You should be constantly testing on your own experimentation systems. Experimentation teams can’t be overprotective of their own processes ”.

---

You might also enjoy these articles:

• Lukas Vermeer - A Global Leader in Online Experimentation

• Strengthen Your Strategy With Experimentation

• Be Hypothesis Driven, Not Idea Led

• Six Lessons From The Award Winning bol.com Experimentation Program

• Create Communities of Practice to Scale Experimentation Faster

• Yelp: A Story Of Building An Experimentation Program

Introduction

Matt Mulvey is a Digital Experimentation Specialist at British Airways. With revenues of €3.5B, British Airways is the UK’s flagship airline, with a fleet size of 267 planes, flying to 206 destinations.

His expertise includes optimisation of digital products through A/B Testing, Conversion Rate Optimisation, Machine Learning and Digital Analytics.

Matt has previously worked as a Senior Conversion Rate Optimisation Specialist and Conversion Rate Optimisation Specialist at Woolworths Supermarkets, New Zealand.

In these roles he was responsible for driving experimentation program strategy and execution, championing experimentation company-wide, and increasing experiment velocity and quality through program optimisation. Prior to this, Matt was a Digital Analyst, conducting A/B testing to improve customer experience.

Originally from Auckland, New Zealand, Matt also established the Analytics & Optimisation Auckland Meetup Group, New Zealand’s first experimentation focussed meetup group, growing to 100+ members.

Listen to this podcast episode on The Experimentation Masters Podcast

In this article we discuss the following:

1. Scaling experimentation in New Zealand’s largest supermarket chain

2. Communicating the experimentation narrative

3. Value and speed to action trumps the scientific process

4. The £7B British Airways transformation project

5. How British Airways organises for experimentation

6. Why your experimentation program is never finished
   
   
   
   
   
   

1. Scaling experimentation in New Zealand’s largest supermarket chain

Growth and scaling of any successful experimentation program tends to thrive when there are two-way forces acting:

1. Strong top-down leadership support

2. Organic, bottom-up grass roots movement

Strong top-down leadership support

The support of experimentation sponsors is invaluable.

In the case of Woolworths, New Zealand’s largest supermarket chain, the Head of Digital had previously done experimentation in a former role.

He was a true believer in the cause and understood the benefits.

“ He was always pushing the Product Managers in the team to be testing, running more tests, and driving it from top-down level ”

 Additionally, when resources were required to grow the program, the Head of Digital was supportive in provide more funding.

Without senior leadership support it is very difficult to develop a culture of experimentation in any business.

Organic, bottom-up grass roots movement

Getting initial footholds is critical to gaining traction with experimentation.

Developing early case studies to showcase results and benefits is an effective way to spread the experimentation message across the business.

Program growth at Woolworths was evolutionary.

“ We would work with one product owner and get them on board. They would then tell another product owner. When product owners moved teams, they would establish experimentation in their new team. This process of evolution slowly increased maturity, spreading throughout the business ”.

Over time, many product teams were able to perform their own experiments independent of the Centre of Excellence team.

2. Communicating the experimentation narrative

When experimentation is new (and foreign) to teams, you need to be extra mindful of how you are communicating.

If people feel like experimentation is a chore, or something they need to complete on a checklist, they may do it, but it's going to be low quality.

They’re not going to be very invested.

It’s important to clearly communicate the value and benefits of experimentation to create motivation and excitement.

“ Experimentation can be threatening. You need to be very mindful about how you communicate what you’re doing, and why you’re doing it. Often the teams you’re working with don’t share the same enthusiasm and passion for experimentation ”

You need to tailor the narrative to your audience, being clear on the WIIFM.

Remember, tailor communications to the audience, not the experimentation team.

Any change can appear threatening if it isn’t communicated correctly.

Tips for pitching to your audience:

1. What is their current reality?

2. What adversity or struggles are they experiencing?

3. What is their desired outcome or vision state?

4. How will your solution create a new reality? (E.g. decrease friction in their life)

3. Value and speed to action trumps the scientific process

There will be times when speed to action must be prioritised over the scientific process.

It’s very easy for experimentation to descend into a purist, academic pursuit.

While it’s always important to ensure the highest quality standards for all experiments, there is a balancing act involved.

“ Experiments always sit higher on the Hierarchy of Customer Evidence. Performing an experiment at 80% confidence is still superior to seeking expert opinion or guessing ”

For many reasons, not all businesses can consistently perform experiments at 95% confidence.

Sometimes you’ve got to go with best data you have available or risk Analysis Paralysis.

There’s an Opportunity-Cost with every business decision.

If you’re enjoying this article, listen to the full conversation on the Experimentation Masters Podcast

4. The £7B British Airways transformation project

British Airways positions as a premium airline, competing against other premium carriers (Air France), rather than budget, low-cost airlines – EasyJet and Ryan Air.

Customer sentiment and, insights from macro research, suggests that British Airways is currently not delivering on the premium airline promise.

The focus of the £7B transformation project is to ensure that British Airways can fulfill and deliver on its customer promise.

The transformation project will invest heavily in the digital estate – Web, App, Performance, Infrastructure.

“ At a minimum, the airline wants to ensure that customers can perform their desired actions (book a flight) without technical challenges “

Website stability and load times must be fixed.

For many years, the British Airways website has been plagued by performance issues and outages.

The opportunities for experimentation are endless through the end-to-end customer journey – checking in at the airport, bag drop, in-flight service and arrival at destination.

5. How British Airways organises for experimentation

Experimentation is not unified through an overarching strategy or leadership role who is responsible for coordinating physical / digital experimentation across the organisation.

There are pockets of experimentation occurring across the physical customer experience and digital (web/app)

In the digital space, a Centre of Excellence (CoE) Model is used - a core team oversees the digital experimentation program, primarily made up of analysts or specialists.

The CoE team are deployed throughout various tribes and product teams across the organisation.

The intent is to embed directly in teams so that the experimentation function can develop domain expertise and deep understanding of customer problems.

6. Why your experimentation program is never finished

Experimentation teams shouldn’t be overly protective of their own processes and systems.

It would be hypocritical to only expect to test on other business team processes, and not the way experimentation operates.

Every experimentation program should be experimenting on its own experimentation processes.

There is no such thing as a finished experimentation program.

In the same way that a website is never finished, your experimentation process will always be an unfinished product.

It’s not like completing a university assignment … when the assignment is handed in it’s done.

You can never rest on your laurels.

If the experimentation program slows your release cycle down too much, then you need to improve the experimentation flywheel to make it spin faster.

To recap

Ongoing experimentation across digital and physical customer experiences will be essential for British Airways on its large-scale, transformation journey.

Experimentation will be mission critical for British Airways if it wishes to wind back the clock to being one of the world’s leading, premium full-service airlines.

Some learnings from the British Airways journey: 

1. Communicate in a non-threatening way to increase support and interest around experimentation

2. Sometimes you’ve just got to go with the best available data or risk analysis paralysis

3. Ensure you digital estate enables you to deliver on your customer promise

4. Embed experimentation specialists in tribes/teams to develop deep domain knowledge

5. Don’t be a hypocrite - experimentation processes should be tested on too

---

Need help with your next experiment?

Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.

Are you struggling with experimentation in any way?

Let’s talk, and I’ll help you.

---