Experimentation Tips From Australia’s #1 Online Real Estate Marketplace
“One of the biggest mistakes that an organisation can make when they’re getting started is trying to run before they can crawl. Trying to go too big, too early. Start small, start localised with a small dedicated team with a focus on one to two key metrics.”
You might also enjoy these articles:
Introduction
How do you become number one at experimentation? It’s a bit of a nebulous question — after all, how does one exactly define being the best at something? — but for businesses looking to stay ahead of the competition, it’s also a necessary one.
And while there’s never a clear-cut path to becoming the best, there are certainly ways for companies to succeed through testing new ideas and bringing innovation up to scale in bold, groundbreaking ways.
One such organisation bringing experimentation to scale on a global level is the international real estate advertising firm REA Group, an Australian-based business that now holds a presence in Asia, Europe and North America as well.
I recently spoke with REA Group Head of Product James Slayter about how his organisation, which has grown to become Australia’s leading online real estate marketplace, tapped into a well of experimentation to adopt innovative solutions and find success.
In order to reach that standing, Slayter consistently asks himself questions such as: how does data inform good experimentation? How can companies innovate while also keeping operations streamlined? And, perhaps most importantly: how do you take everything you’ve learned, internalise it and hone it to become number one?
In this article we discuss:
How to keep things simple
What a good hypothesis looks like
How you lead with experimentation
What experimentation looks like in practice
Why it’s OK to fail
1. Question one - How do you keep things simple?
Slayter believes that when it comes to scaling experimentation, the first (and most important) rule is to keep things simple.
Experimentation can get very complex, very quickly, especially when you think about the statistics and mathematics that sit behind every experiment. Bringing experimentation up to scale on a national, international or even global level too quickly can create challenges.
Slayter suggests that one common mistake people make when getting started with experimentation is to try and measure multiple metrics at the same time.
The more you can focus your experiments on one key metric it will increase the effectiveness of your experiments.
“One of the biggest mistakes that an organisation can make when they’re getting started is trying to run before they can crawl. Trying to go too big, too early. Start small, start localised with a small dedicated team with a focus on one to two key metrics.”
2. Question two - What does a good hypothesis look like?
In addition to measuring experiments with a single metric, Slayter says it’s also important to spend time getting your hypothesis correct.
Really thinking about what you want to achieve — before you go out and try to achieve it — will help create the framework required to run a great experiment. Failing to do so can lead to unintended consequences: a poorly developed hypothesis might lead to experiments that yield incorrect or misguided answers, the results of which could put your business on a path toward poor conversion rates, sluggish growth and more.
“It's about sitting down and thinking about what you want to achieve and having a really good hypothesis statement in the framework of ‘IF [product change], THEN [outcome], BECAUSE [research/insight]” says Slayter. “And then always think about secondary metrics. So it's not just about testing the one thing — obviously keep it simple and use one metric — but definitely look at how it impacts some of the other metrics around the experience. You want to make sure that you're not increasing conversion in one space, but then actually decreasing conversion or user experience in another space.”
So: you don’t want to test for one metric and then implement a change that ends up decreasing potential in other areas.
But how do you work around these kinds of technical issues?
The answer, according to Slayter, is to anchor your hypotheses in solid research. After all, it doesn’t pay to come up with an educated guess — such as changing your website colour scheme or creating a different user interface — simply because you think customers will be drawn to those changes.
“Using research or an insight, or some analysis that leads to that hypothesis statement.... that part of the hypothesis is quite important,” says Slayter. “[A bad hypothesis might be], ‘if we change this button colour, then we'll see a 5% uplift… because we just think people will like that.’ You can test to see if a new button colour is better, but it’s best to have an insight or a piece of research tied to that test.”
3. Question three - How do you lead with experimentation?
Does experimentation help company heads lead effectively?
The answer is both yes and no: if you’re able to enable data-driven decisions that come as the result of good experimentation, then the answer is unequivocally yes. If you’re not currently a metric-driven business, and you’re used to relying on the highest paid person at your organisation to make decisions, then probably not.
“I think that probably one of the big things that experimentation enables more broadly, and it also comes down to leadership too, is being able to enable data driven decision making”.
For metric driven businesses, experimentation takes the opinions out of decision-making. It enables businesses to make decisions based on North Star metrics or tangible metrics that can be influenced and optimised.
“Experimentation empowers product managers, designers, analysts and business people to really challenge existing opinions and challenge the status quo a bit,” says Slayter. “And I think that it definitely helps leaders in that, you've got more people that are contributing to the decision making process.”
That means great experimentation is good for democratically made decisions — which, in most cases, are beneficial to companies that might otherwise rely on siloed decision-making tactics.
Experimentation enables clarity in leadership.
If you’re enjoying this article, listen to the full conversation on the Experimentation Masters Podcast
4. Question four - What does experimentation look like in practice?
It’s easy to talk about scaling experimentation. But how does it happen in practice?
One way REA Group has built their experimentation practices into everyday operations involves adopting a regular showcase to document results. In other words, Slayter’s company takes time to deliver the results of experiments through demonstrations that explore the process step-by-step.
“So it's not just a showcase of the results in the experiments,” says Slayter, “but also an opportunity to bring in guest speakers or to bring an expert in the field, to actually then talk through, how they do experiments and really create a bit of momentum and a forum around it, get people to share their experiences.” Doing so gives everyone insight into what’s being tested, why it’s being tested, and what changes need to happen as a result of the experiment at hand.”.
And, another thing .. try and keep your experimentation governance processes lean and mean. Try to remove as much governance as possible without compromising the quality of your experiments or unnecessarily introducing business risk.
5. Question five - Is it OK to fail?
Slayter says showcases are also an important way to start a conversation around a necessary subject most businesses don’t want to touch: failure.
“You know, I think showcases also create a bit of a safe place as well. The experiments, a lot of them fail. And it’s important to be showing people that failing is good in experiments, because often, an experiment is much cheaper to set up, and much cheaper to run. So I think, for me personally, and I've seen time and time again, that people learn more from failed experiments than they do from the successful ones.”
That’s right: organisations that embrace failure might actually get more out of experiments gone wrong than the ones they get right.
Market leaders like Amazon, Netflix, Apple, Google and others often take a fail-forward approach of testing and experimenting as much as possible to stay ahead of the curve — even if that means a majority of what they try doesn’t necessarily work.
This shows up again and again in the parts of these businesses that, while ultimately unsuccessful, led to greater heights down the road: Amazon began as a bookseller and expanded into commercial retail; Google created failed AR visual technology (does anyone remember Google Glass?) only to implement that technology more fully through the successful Google Earth; Netflix moved on from its soon-to-fail DVD mailing strategy in order to become the world’s largest streaming platform.
“As with most businesses, it's unnatural to make failure okay. But it's definitely something that we're working on as an experimentation community within REA Group. Thinking more broadly within REA Group, probably like a lot of businesses, the Netflix's of the world, and the Amazons, and the Facebook's, we’re trying to really celebrate failure.”
Experimentation is central to increasing the adoption of failure. It’s a great way to highlight that failure is OK, and to be bold with your experiments. If experiments don’t work, then it’s a really good insight for everybody else.
Become number one by asking the right questions
It’s clear that becoming the most successful at what you do doesn’t happen overnight. It’s a marathon, not a sprint.
The process requires asking great questions, streamlining your testing metrics, anchoring your experimentation in research and much more. At the same time, it’s still important to strive for the kind of scaling that Slayter engages in every day. Doing so will make your business a leaner, better, more innovative environment for everyone involved.
Curious to know more ways to successfully experiment in your own business setting?
Listen to the full discussion on the Experimentation Masters Podcast between Slayter and I to learn more tips from Australia’s number one real estate advertiser.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.