How Sky Established And Scaled Experimentation
“If you’re not testing, there’s a really good chance that 90% of what you’re doing is a complete and utter waste of time and money — or, worse, is actually having a detrimental impact on your business.”
You might also enjoy these articles:
Introduction
I had the opportunity to talk with Journey Further Conversion Director Jonny Longden, who spoke about his experiences as Head of Experimentation at Sky.
Longden, an experimentation professional with a rich background in data science and analytics, believes that experimentation is key to creating the kinds of products and services that cut through the clutter and reach their target audiences.
During our conversation, Longden provided practical advice about how to implement and scale experimentation in a large, complex organisation. Keep reading to learn some of the hard-won lessons Longden has learned from a lifetime of successes, failures, and insights from experimenting at the coalface.
In this article we discuss:
Why you should test everything
Why experimentation needs to be the heartbeat of your business
Why corporate culture makes experimentation challenging
Why experimentation isn’t about winning or losing
Why you should be wary of competing interests
Why communication is critical for cultural change
1. Lesson one - Test everything
Before brands engage in experimentation, Longden wants to make one thing clear: there’s no one-size-fits-all secret to proper testing principles.
Once that’s understood, it becomes much easier to start experimenting in a way that’s beneficial to your business — instead of process-driven and superficial.
“After fourteen years of running tests on websites, it would be really easy for me to go around saying, I know what you should do to your website, because I have all these best practices… and it’s simply not true,” says Longden. “The only true thing I can say after running fourteen years of tests is that you just have to test everything.”
In fact, the only thing Longden says you absolutely need to do is run tests. Without them, you might be doing more harm to your business than good.
“Any company that asks for advice, I tell them the same thing: if you’re not testing, there’s a really good chance that 90% of what you’re doing is a complete and utter waste of time and money — or, worse, is actually having a detrimental impact on your business.”
2. Lesson two - Experimentation is the heartbeat of your business
Another problem is the way businesses perceive experimentation. Experimentation is seen as another channel, like SEO or Paid Advertising.
Experimentation is the heartbeat of your business - it’s the operating model of your business. Experimentation should not be an addendum, some add-on tool or method. Experimentation is how your business develops new products, make decisions, invests, and develops strategy.
“At the heart of it, it’s just about using research and data to come up with hypotheses, and finding ways to validate those hypotheses in real situations with real customers. And why wouldn’t you do that? That’s how all the great things in the world have been brought about — through the careful application of the scientific method.”
Longden believes marketing and business experiments should be no different than the tests used to discover other great innovations that have come about in the last century: from spaceflight to medicine, people get incredible results when they ask questions, rigorously apply their hypotheses to real-world situations and adjust for the results.
“If I could just get people to understand that,” Longden says, “we could get them to see beyond all these slight misperceptions and strangeness that comes with [marketing-speak].”
3. Lesson three - Corporate culture makes experimentation challenging
Another reason there’s so much resistance to experimentation in business is that the approach of experimentation is fundamentally opposed to how organisations are run. There’s a culture clash!
Most of this comes from the fact that people in large corporates are hired for their experiences, opinions, and past successes, rather than for their ability to exercise disciplined thought processes. It’s the commodity in the business world.
“When people get hired into a company, they have to justify their value, and what they’re being paid is related to their past experience… rather than their ability to exercise a thought process,” says Longden. “And everyone wants to bring their own opinion and their own ego and mind to the table. That’s fundamentally what underpins business culture, and that’s why experimentation is so challenging a lot of the time.”
In other words, the phrase “I don’t know” gets lost in a business setting, because to not know something is considered taboo.
Leaders are expected to know exactly what they’re doing, and the prospect of not knowing — with the potential of failure attached to it — is looked down upon by colleagues and shareholders alike.
This leads to a vicious cycle: by not testing new ideas, and always relying on tried-and-true wisdom and practices, businesses end up guessing, and fail.
It’s only by decreasing the stigma of failure that companies can change, grow, and succeed. “If people think they’re responsible for an initiative that fails, and they fear they might get fired… that causes people to either avoid doing things that they think might fail, or to spin things that they have done into something successful.”
That last notion is worth highlighting – often times, businesses will cover up past failures and use internal PR tactics to sugarcoat failed ideas and turn them into pseudo-successes. Did you know that some businesses have never failed?
While that might be good for a company’s public image, it ends up being unhelpful, because it removes the ability of an organisation to learn from its own mistakes.
Rather than focusing on failure, reframe thinking in your organisation to focus on the upside of experimentation – organisation learning. All experiments should result in organisational learning.
The faster your organisation can learn, the faster you can make decisions, which means you can create and deliver value for your customers fasters. This is a true competitive advantage.
If you’re enjoying this article, listen to the full podcast conversation on Experimentation Masters
4. Lesson four - Experimentation is not about winning or losing
The remedy to all this anti-failure behaviour, says Longden, is to adopt a childlike mentality to experimentation. When children learn they engage in trial and error. Falling off a bike is how you learn to ride a bike.
Accepting failure and reframing it as a learning experience rather than a losing experience can help companies better stomach failure — if they know the process of failing will lead to success down the road, it might be easier to move forward with experimentation.
A large portfolio of small experiments will always outperform a small portfolio of large trials.
“People talk about winning and losing in experimentation, and that language in and of itself is very charged… like, who wants to lose?” says Longden. “But if you went and asked a scientist who was trying to develop a cancer cure what the outcome of their experiment was, there’s no way they would say their results win or lose. It just doesn’t make sense. You can’t win or lose in experimentation, you’re finding something out, and that’s the whole point.”
In a lot of ways, business can be more juvenile than a child.
5. Lesson five - Bringing experimentation to life at Sky
Longden, when Head of Digital Experimentation at Sky, learned several lessons from his time at the digital streaming and news organisation.
While the company was already dabbling in experimentation before he joined, the approach was fragmented and lacked standardisation, leading to mixed results. There was no culture or cohesive strategy for experimentation.
When hired, Longden worked to embed experimentation into Agile ways of working. This was a unique opportunity to develop pioneering ways of integrating experimentation into product development, without the hang ups of legacy processes and procedures.
“I mean, the real beauty of it was that we were building this whole kind of environment from scratch,” says Longden.
The first step was hiring good people. “I managed to hire a really good team of people, who were skilled and capable of running experimentation”.
The experimentation team were aligned to different squads in the business (Sky News, Sky Sports), embedded in the scrum teams. It was important that the experimentation team worked closely with product owners to build a strong team culture.
Longden suggests, “product teams don't want somebody in experimentation throwing things over the wall at them. They want to be able to work together and discussing things together”.
6. Lesson six - Be wary of competing interests
Experimentation doesn’t operate in a vacuum. If an experiment tests positively, it doesn’t mean that it will be implemented. You still need to lobby and influence to ensure that value is delivered.
“We were coming up with successful outcomes of experiments that needed to be pushed into production, and those things would just end up sitting in a backlog and never actually getting done”.
One of the first things Longden noticed at Sky was how engineers were focused primarily on two things — fixing product bugs and tackling new, big product feature launches — instead of putting their energy toward experimentation.
The result was that products worked well, and new launches were happening regularly, but engineers weren’t dedicating enough time to innovation.
To succeed, Longden had to uncover why experimentation was falling by the wayside. What he learned was that the engineering team focused on other areas because they were more rewarding than experimentation.
“When there’s a critical bug, everyone wants to avoid brand damage, so those issues immediately get solved… and at the opposite end of the scale, everyone wanted to work on those bigger, more complex projects because those projects tend to be more interesting for developers to work on. And by nature, what comes out of experimentation tends to be simple to do, so that sort of stuff never gets done.”
Instead of telling the team to stop focusing on the things they readily gravitated toward and enjoyed, engineers were required to split their priorities across the three areas: bug fixes, projects, and experimentation. The key part? The engineers would rotate regularly through each focus area to ensure work diversity and career development.
7. Lesson seven - Communicate to effect cultural change
An effective experimentation program is high on organisation communications. Constant stakeholder engagement and communications are required to drive change through complex, political, matrix organisations.
“One of the most important things that we were able to do in terms of developing the culture was trying to surface and socialise the bigger and more high-profile tests and their outcomes”.
It can be tough to quantify the value of an experimentation program. Longden suggests that the best way to get traction is to focus on the commercial impacts of the program.
“That’s really what ended up being the glue between what we were doing in the product environment, the rest of the business and the decisions that were made. It was being able to show the commercial impact, quite clearly, of what we were doing”.
Final lesson - Establish your own experimentation program
What we’ve learned from Longden’s experiences establishing experimentation at Sky is that implementing a large-scale experimentation program can provide countless benefits — provided businesses are willing to persevere through the try-fail process and embrace the process of questioning everything.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.