Strava - How To Use Negative Experiments To Create More Customer Value
“Experimentation is the tool that we use to measure changes in our product for users, user journeys and user experience. Experimentation isn’t an end game or goal. We rely heavily on experimentation to understand how the changes we’re making on the product impact outcome metrics like retention and subscriptions.”
You might also enjoy these articles:
Six Lessons From The Award Winning bol.com Experimentation Program
United Nations: Solving The World’s Toughest Challenges With Experimentation
Introduction
In this blog post, I chat with Jason van der Merwe, Director of Engineering and Growth at Strava about how he uses negative experiments to create more customer value, and better user experiences.
With a background in iOS development, Jason manages multiple engineering teams at Strava, being responsible for the Growth, Insights and Optimisation teams, ensuring operational efficiency and cross-functional collaboration across Strava teams.
Strava is the popular exercise tracking and social networking service, with more than 110 million users and 8 billion activities uploaded.
What is a negative experiment? A negative experiment removes elements of the user experience to measure impacts on customer behaviour.
Negative experiments are a great way to dispel organisational assumptions, identify new value creating opportunities and amplify customer experiences.
Jason shares his practical tips and lessons on how Strava perform negative experiments.
In this article we discuss:
Strava’s experimentation journey
Experimentation culture at Strava
Strava’s biggest challenge with experimentation
Using negative experiments to create more customer value
1. Strava’s experimentation journey
(A). Getting started
When Strava were starting out with experimentation, all they knew was that they need to start performing experiments.
“The hardest part about getting started is learning how to experiment.”
Initially, experimentation is not easy. There are many considerations - experiment design, customer segmentation and cohorts, evaluating and analysing experiment performance.
And then when you start performing those early experiments, you start questioning user behaviours and actions.
In the early days, Strava was performing simple experiments around the new user experience. For example, getting users to add a profile photo and complete profile information.
Over time, as the team started to perform more experiments, they were able to get clearer on key focus areas and growth drivers.
(B). Unlocking the “A HA” moment
The Strava growth team regularly sought counsel and advice from growth experts at market-leading companies, including Facebook, Twitter, and Dropbox.
One of the big quantum leaps for the Strava growth team was understanding what the user activation moment meant.
The user activation moment (aka “a ha” moment or “magic” moment) is where you get a user to perform a certain action or reach a certain milestone in their experience. When a user completes this milestone activity it is highly correlated that the user will stick around longer and become more engaged.
For new Strava users, the team identified the activation moment as one activity upload and one follow in the first seven days.
The activation moment was defined through 1). User research – looking at what users are trying to accomplish in the product and 2). Correlation analyses – looking at what user actions were highly correlated with retention (one week, two weeks, four weeks, six months etc.)
“Once the activation moment was defined, the growth team performed hundreds and hundreds of experiments to try and reduce friction for users in uploading an activity and following friends.”
Making the activation moment easier and frictionless was a key focus of the Strava experimentation roadmap for a long time.
(C). Increasing experiment velocity
Through a long-term, specific focus on improving the user activation moment, Strava was able to get better and better at experimentation.
The team were able to move from hypothesis > design > execution > analysis faster.
“Over time, you just get better at experimentation when you do it a lot.”
Conducting more experiments and decreasing experiment cycle time presented new organisational challenges, increasing friction in the experimentation flywheel.
With so many experiments running concurrently, coupled with a moderate audience size, the growth team reached an upper limit for experiments, running out of users.
(D). Improving experimentation maturity
One process improvement that the Strava team implemented was a test brief. The purpose of the test brief was to standardise the way inputs into experiments were documented.
“Strava were constantly experimenting on their experimentation processes to make experimentation easier and more efficient.”
The growth team were also constantly experimenting with the most effective ways to communicate experimentation results and insights with key stakeholders.
Interestingly, Strava chose to keep experiment documentation simple, preferring to capture experiment results in Google Slide decks. This process was in place for more than six years.
More recently, the growth team have flipped experiment documentation into Confluence to gather more detail and ensure better searchability.
Automation of standard experimentation tasks can be a great way to drive process efficiencies in your program. If you’re perform the same repetitive task over and over, it’s best to automate.
This is what Strava did with their Experimentation Reporting Framework, the tool for evaluating all their experiments. Data Scientists were writing the same SQL query over and over to evaluate experiments, so they created one SQL query (Lord of the Rings) to rule them all.
While the Strava growth team are performing hundreds and hundreds of experiments in a more sophisticated way, the work to keep improving experimentation is never done.
2. Experimentation culture at Strava
Strava’s ideology with experimentation – it’s a tool that’s used to measure changes in the product for users.
Experimentation is not a goal or an end game. Experimentation is the tool the business uses to figure out how to make changes.
“We heavily rely on experimentation to understand the impact on our output metrics, things like retention, or subscriptions, how the changes we're making in our product can impact those metrics.”
For the most part, Strava try to A/B test everything. Almost all product changes are wrapped up into an experiment – new columns, email campaigns, a new checkout page, API performance change.
Strava are always trying to understand how much product changes really impact the user, the user journey and user experience.
Sometimes, there are situations where experimentation can difficult. Strava is a social network, and network interference can cause complications with experiments.
There is a big focus on making experimentation as easy as possible. Teams don’t want to spend a lot of time setting up and executing experiments.
Launching an experiment shouldn’t be more difficult than making the production product change.
“A big part of Strava’s experimentation culture is trying to improve the experimentation platform so teams can run tests without even thinking about it.”
3. Strava’s biggest challenge with experimentation
One of Strava’s biggest challenges with experimentation is the relationship between input and output metrics.
Many experimentation programs struggle with this.
Output metrics are strategic business objectives such as Subscriptions and Churn. Input metrics are the short-term, driver metrics used to measure experiment performance – feed views, activity uploads, accepting push notifications, viewing heart rate data etc.
Experiment input metrics should ladder up to strategic business objectives.
For example, we might think that users viewing heart rate data is a great input metric. If user heart rate data views increase, there is a probability that subscriber retention will increase.
“Input metrics are far more sensitive to change than top-line strategic output metrics. Very big shifts are required (double digit %) to input metrics to observe a micro positive change (1%) to a strategic output metric.”
This scenario is unlikely.
Therefore, Data Scientists perform a lot of correlative analysis to understand which input metrics most representatively link to output metrics. The big challenge for all businesses, not just Strava, is that it can be very difficult to prove causal relationships from input metrics to output metrics.
Because an input metric is correlated to an output metric it’s not necessarily the reason why Subscriptions are increasing.
Being in the middle ground and, assuming your input metrics link causally to output metrics, is the death zone.
If you’re enjoying this article, listen to the full conversation on the Experimentation Masters Podcast
4. Using negative experiments to create more value
(A.) Overview
As previously mentioned, Strava’s user activation metric was one upload and one friend follow in the first seven days.
While the upload component of the activation metrics hasn’t changed, the “follow” element has changed over time.
People are becoming more sceptical to follow someone in the first seven days. As a result, the importance of this metric has decreased.
“If I was giving advice to another company … you should never have an activation metric that has two prongs to it like we did.”
One of the things that Strava has been doing to try and prove causal relationships between input and output metrics is Negative Experiments.
(B). What are Negative Experiments?
By definition, a Negative Experiment is a type of A/B test where you purposefully remove an element or component of a user experience and measure the impact on user behaviour.
For example, Strava added friction in the user onboarding process by making it more difficult to follow friends. The team wanted to understand how this change impacted Retention. Follow Rate and Retention both decreased. However, Retention only decreased marginally. Therefore, there was a lot less sensitivity between Follow Rate and Retention than expected.
It can be hard to improve a metric, however, it’s much easier to make an experience worse, and decrease a metric.
(C). Why perform Negative Experiments?
A product, user experience or feature can become complex and bloated over time due to the addition of many layers of product change. Products are built in piecemeal fashion and can easily become over-engineered.
Additionally, product experiences and product ecosystems decay over time.
What worked at one point, may not be effective anymore. It can be difficult to understand what is truly driving value for users in your product.
In Strava’s case, performing the Negative Experiment provided a big “unlock” for the new user activation team, generating new and unexpected insights. Assumptions that were relevant from many years ago no longer held strong.
The key insight for the team was to focus more on the activity Upload moment in the first seven days, and community building (friend follows) at a later stage in the user journey.
(D). Benefits of Negative Experiments?
Once you get to a point where your user experience is getting complex, and there's a lot of elements to the core product, it can be difficult to understand what is truly driving value for users.
The Negative Experiment is a great way to stand back and remove some components of the user experience and see what happens to user behaviours.
There are five key benefits to the Negative Experiment:
Helps teams to understand which elements of the product drive user value
Helps teams to overcome status quo and challenge organisational assumptions and beliefs
Helps to highlight improvement opportunities in the user experience
It’s easier to rebuild your product from a simplified starting point
Helps teams to better understand causal relationships between business metrics
In so many instances, people just assume that the product works.
It’s much harder to throw away what you have, go back to the drawing board, and try something completely different that may have bigger impact.
Negative Experiments are a great way to test everything, versus nothing, retrospectively.
(E). Set boundaries in place to reduce business risk
Negative Experiments are no different to any other A/B test. You need to have clear experimentation guardrail metrics in place to monitor and measure the impacts of experiments.
Be prepared to pause or stop experiments if guardrail metrics are negatively impacted, particularly when conducting monetisation or revenue tests.
Enable teams to use their own judgement and decision-making, however, ensure that you have safety nets in place to provide people with a soft landing.
In summary
Products and user experiences can become complex and unwieldy. It can become difficult to distinguish which elements of the product are value creating, and which aren’t.
Product and growth teams should be constantly questioning everything.
Negative Experiments are a great tool to help teams challenge status quo and uncover new value creating opportunities.
Some practical tips for performing Negative Experiments:
Perform Negative Experiments before additive experiments
Allow teams to use their own judgement and decision-making
Exercise caution when performing Negative Experiments on revenue
Ensure guardrail metrics are in place to monitor experiment performance
User experiences decay and age with time.
What worked years ago may no longer be effective.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.