How To Structure Your Experimentation Team

---

Experimentation redefines cultural norms. Data becomes central to every business decision. Experimentation shifts your business culture to be more objective and collaborative.

---

You may also enjoy these articles:

• Use Experiments to Get Your Aim, Before Firing The Cannonball

• Strengthen Your Strategy With Experimentation

• Be Hypothesis Driven, Not Idea Led

• Overview of Business Experimentation

• Sequence Experiments to Decrease Risk and Uncertainty

Introduction

Experimentation has a powerful and transformational effect on businesses, completely redefining how problems are solved, decisions are made, and capital investments allocated.

However, scaling an experimentation program in a larger enterprise can often be one of the toughest challenges, even for a high-performing organisation. Widespread cultural change is never easy, taking consistent effort and time.

Every company has existing business workflows, governance processes and funding models that experimentation touches and is required to integrate with.

Experimentation gets right at the heart and core of how a business operates.

Establishing a dedicated experimentation team can be the best way to drive sustainable, long-term success with experimentation.

Leading experimentation programs commonly originate through a centralised function. Embedding a culture of experimentation requires a coordinated and systematic approach to drive tangible outcomes.

Overview

Implementing an experimentation program in any business takes time and requires long-term commitment to realise the full benefits and potential of the program.

“Scaling an experimentation program can be one of the toughest challenges for any business”

Experimentation completely redefines cultural norms, ways of working, workflow processes, governance models, decision-making and the way business capital is utilised and allocated.

Rather than trying to re-engineer key functions of your organisation, it’s always prudent to understand how experimentation can fit within existing business processes and delivery models.

This way, you can get an experimentation program up and running faster, generating momentum and results quicker.

Having dedicated resources, operating processes and funding is essential to making scaled experimentation a reality in your organisation.

Achieving success with experimentation requires a highly systematic and coordinated approach. It’s not something that can succeed with an ad hoc commitment.

Implementing an experimentation program may seem a bit daunting to begin with, however, it’s worth the investment.

“Shifting business culture from subjective to objective creates a more collaborative and healthier culture”

Data should be at the heart of every business decision that’s made.

Every decision that you make needs to have an impact on customer experience and business performance.

All businesses should be ensuring that they’re spending every dollar in the most efficient way.

Implementing experimentation in your business

An important thing that you need to appreciate before implementing any experimentation program in your organisation is that every business is different.

What has worked successfully in one organisation, may not work in your organisation.

Don’t feel that you need to follow the recipe to a tee.

Every business has different business processes, governance models and risk appetite.

Take the below structures as a starting point, rather than an endgame.

You’ll need to understand how and where experimentation can fit within your existing delivery models and development frameworks.

An initial experimentation model is just a north star to get going. Think of it as a guiding principle more than anything.

Don’t get too hung up on the model.

Expect this model to adapt and evolve as your experimentation program matures and organisational needs change.

“What’s most important is shifting cultural norms and how people think so that experimentation can be embraced. These are the things that really matter”

Three types of experimentation team structures

There are many different ways that you could potentially structure experimentation in your business. Each approach comes with advantages and disadvantages.

Invariably, the way that you setup the experimentation team will influence how the program runs.

The three most common types of experimentation operating models are:

1. Centralised Model

2. Centre of Excellence Model

3. Decentralised Model
   

Research by Optimizely suggests that, “47% of businesses use a Centralised Model, while the remainder use a Decentralised Model (27%) or outsource”.

As most organisations mature it is common to transition from a Centralised Model to a more decentralised model.

SaaS businesses typically prefer to operate using a Decentralised Model, while larger enterprises lean towards a Centralised Model.

1. CENTRALISED MODEL

From my experience, a centralised model is a great way to kick-off experimentation in an organisation and get some momentum going.

In this model, the centralised experimentation team effectively operates as a shared services provider, conducting experiments for the business units and teams.

This is the simplest model.

It’s a good way to start small, generate positive impact and benefits, deliver value, before increasing the breadth of experimentation in the business.

The experimentation team does not own strategy or roadmaps. These responsibilities are owned by the business teams.

The experimentation team is responsible for the development, implementation and maintenance of experimentation tools and platforms.

All of the necessary resources to manage and run experiments are located in the centralised team.

Centralised Model

Roles and responsibilities

In the Centralised Model, a dedicated experimentation team is responsible for the following:

• Develop a culture of experimentation in the organisation

• Lead experimentation best-practices in the organisation

• Conduct ongoing training and capability uplift

• Facilitate execution of experiments with business units

• Share learnings and insights from experiments

• Experimentation program communications

• Measurement and tracking of program performance and key metrics

Workflow management:

Experimentation workflow management for the Centralised Model.

Benefits
The reason that the Centralised Model is popular with many organisations is that this approach has many benefits:

1. All experimentation data and results are stored centrally

2. Easy to maintain alignment on experimentation vision, strategy and objectives

3. Strong rigour and compliance around experimentation methods and processes

4. More cost effective due to one shared technology toolset / platform

5. Promotes open sharing of data and insights

6. Avoids vested interests, competition and turf wars
   
   

Challenges
Some of the key challenges with a Centralised Model include:

• Cross-functional collaboration and stakeholder engagement

• Potential for bureaucracy to stifle efficiencies

• Data integrity being questioned by stakeholders

2. CENTRE OF EXCELLENCE MODEL

The Centre of Excellence (CoE) model is basically a hybrid model, combining elements of the Centralised and Decentralised models.

In this model, some experimentation resources are retained in the central function, while others are distributed out into the business units, teams or customer journeys.

The CoE is still responsible for providing oversight on experimentation methods, tools and platforms. The team plays a strong stewardship role, ensuring that guidelines and principles are adhered to.

The key difference between the Centralised Model and CoE Model is that the distributed teams are responsible for analysis of experimentation data.

Centre of Excellence Model

Workflow management:

Experimentation workflow management for the CoE model.

Benefits
Some of the benefits of the CoE model include:

1. High experimentation standards and compliance are maintained

2. Provides business units and teams with more flexibility

3. More trust in data analytics

4. Potentially a catalyst for cross-functional unification
   

Challenges
Some of the key challenges with a Centralised Model include:

• Blurring of lines with responsibilities

• Funding for experimentation program expansion
  
  
  

3. DECENTRALISED MODEL

In the Decentralised Model, experimentation resources are spread across various teams in the business.

These expert teams typically have a specific focus, commonly aligned to an element of the sales and marketing funnel – Acquisition, Onboarding, Conversion, Retention, Engagement etc.

The teams develop a very deep understanding of the customer and problem spaces.

If the business is seeking specialisation and deep expertise in specific areas, a Decentralised Model is superior to a Centralised Model.

However, the trade-off with a deep level of specialisation can be short circuiting knowledge sharing and communication with other teams across the business.

In this model there is no centralisation of experimentation resources in one team or department.

Experimentation design, execution and analysis is performed by resources within the respective business branches.

This model is common in SaaS businesses.

Decentralised Model

Workflow management:

Experimentation workflow management for the Decentralised Model.

Benefits:

Benefits of the Decentralised Model include:

1. Higher level of flexibility and autonomy for business teams

2. Experimentation resources required to develop broader knowledge base

3. Advantages with understanding of customer insights in local geography

Challenges:

While there is the allure of greater flexibility and freedom operating under a Decentralised Model, there are also some downsides:

• Lack of communication between business teams

• Establishment of “experimentation islands” in the organisation

• Not capturing and sharing organisational lessons learned

• Higher costs due to replication of tools and technology

• Duplication of work effort

• Validity and reliability of data due to undisciplined experimentation processes

• Fragmented and disjointed experimentation culture
  
  

Aligning your experimentation structure to organisational capability

If your organisation is new to experimentation, you need to think carefully about how you structure and kick-off the program.

As stated, the way that you establish your experimentation program will influence how it runs in the beginning.

Expect that your experimentation model will change and adapt over time. It’s not fixed in stone but should be flexible to accommodate changing organisational needs.

One thing that’s really important is to marry up experimentation organisational capability with the structure and model that you select.

Walk before you run

Just like any new capability or skill, it takes time and practice to achieve competency and proficiency. Experimentation is no different.

Developing a really strong experimentation muscle in your business should be considered a journey over time.

It can take many years to develop a high-performing experimentation culture in a business.

Start simple

I recommend a Centralised Model as a good place to start, if funding and business buy-in permit.

A Centralised Model works well when there’s a strong level of organisational and leadership support for experimentation.

The Centralised Model is best for organisations that have a low-level of competency, maturity and organisational capability with experimentation.

Having a centralised locus of control at the beginning of your experimentation journey is definitely helpful.

“It provides a clear vision for the program, a strong discipline and rigour around experimentation methods and clarity and consistency in communications and messaging”

Standards and expertise can be built before expanding into the business units.

A big upside is that there is a dedicated team of experts who can conduct targeted capability build within the organisation to continue developing business proficiency in experimentation methods, tools and mindsets.

There may come a time when your organisation could outgrow the Centralised Model and this approach becomes non-scalable.

One of the drawbacks of the Centralised Model is that experimentation throughput is limited by headcount in the experimentation team.

Continuing to scale headcount in a centralised team may become cost prohibitive for some businesses.

If your organisation is seeking to increase experimentation velocity and scale, which it always should be, you will need to consider transitioning to a more decentralised model.

NOTE:

In my previous role, I implemented and scaled an experimentation program in a leading tier one organisation, running more than 1,000 experiments.

It took nearly three years of sustained effort before the business reached a level of experimentation maturity that implied a readiness to transition from a Centralised Model to a Decentralised Model.

Don’t get ahead of yourself

If you’re starting out on your experimentation journey, I would be cautious about going all in and commencing with a Decentralised Model.

The Decentralised Model requires a high-level of experimentation maturity, capability and tools and infrastructure.

There are some situations where this may make sense though.

“If there’s no broader organisational commitment for experimentation, it may be prudent to establish an experimentation function within a business unit or team”

The advantage of this model is that business teams have a high degree of flexibility and autonomy with experimentation.

The major disadvantage is that there’s no centralised oversight on experimentation quality control.

A great deal of trust and faith is placed in business teams to ensure that they’re running disciplined experimentation processes.

There’s nothing worse than running undisciplined and poorly designed experiments. This results in bad quality data.

This may result in the implementation of projects or initiatives that were based on poor quality or unreliable data.

Conclusion

Experimentation is a really powerful way to transform cultural norms and business problem-solving.

Experimentation can completely redefine the way that a business measures and evaluates customer value, makes decisions and allocates capital.

There are three common experimentation models – Centralised Model, Centre of Excellence Model and Decentralised Model.

The Centralised Model is commonly used in most businesses.

Implementing an experimentation program in an organisation doesn’t come without its challenges. Experimentation intersects with existing business processes, governance models and budgeting processes.

Don’t get too hung up on the model. You’ll be able to figure things out along the way.

---

Need help with your next experiment?

Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.

Are you struggling with experimentation in any way?

Let’s talk, and I’ll help you.

---

References:

CXL, Optimizely