Denise Visser - Lessons From The Award Winning bol.com Experimentation Program

Episode 009 - Denise Visser - Lessons From The Award Winning bol.com Experimentation Program

 

Gavin Bryant  00:03

Hello and welcome to the Experimentation Masters Podcast. Today I would like to welcome Denise Visser to the show. Denise is the captain of Team [email protected]. Bol.com is the biggest e commerce business in the Netherlands, generating 5.5 billion euros in revenue in 2021. The business serves more than 12 million customers in the Netherlands and Belgium, offering more than 34 million products, 47,000 sales partners sell their goods through the ball.com platform. 

 

In this episode of the show, we're going to discuss how Denise established and scaled an award winning experimentation program @bol.com. Welcome to the show, Denise.

 

Denise Visser  00:53

Thanks. Thanks for having me.

 

Gavin Bryant  00:56

Now Denise, we were just chatting briefly off air about your recent holiday to the snow. Do you think that skiing is a good representation of experimentation where you try, you fall down? You fail? And then you try again?

 

Denise Visser  01:15

Yes, but that also represents maybe the different kind of people in the company. So I have a son of 11 year olds, he is more.... Yeah, I could say more humble person. So when he went on the slope, and he makes turns from left to right, left, right, and very easy, easygoing. And he said, 'Yes, I need 10 minutes to go up by the lift, why should I go down fast'. And so he was taking no risk. And then my daughter, she's 7 years old, maybe she didn't run into anything yet. But she decided to go straight down on all the slopes. So her highest speeds is more than 60 kilometers per hour. It was nice to see that both took their own way of learning skiing.

 

Gavin Bryant  02:17

Yeah, interesting perspective, the risk averse versus the risk taker.

 

Denise Visser  02:24

Yes, definitely.

 

Gavin Bryant  02:26

Now Denise, let's start by discussing an overview of your background and your experience. What are those key experiences in your background that have led you to be Captain of Experimentation @bol.com?

 

Denise Visser  02:43

Yes, so I am now Product Manager of Team Experimentation for 3 years, but I work already for over 14 years at bol.com. It is my first job. But I would like to get more bulky introduction, to give you some insight also in my studies... My college days started in 1994. And I studied Business Informatics where you are trained as an intermediary between Managers and IT. 

 

So ecommerce had my interest right then although it was still quite new. Luckily, the listeners cannot see my gray hair. But I saw the ease in which you could bring about a transaction online but saw the complexity and logistics so I decided to also study logistics and economics, which was possible within the same faculty. So it was an easy game. And during the study, I started a company webtastic [phonetic 03:50], with two fellow students and we were web developers. Under the guidance of one of those fellows’ students, I learned how to program, and at the same period, I also went to university for the study Information Science, which focus more on the human computer interaction. 

 

So a lot of education, but I really enjoyed the continuous learning and the flexibility and responsibility of having my own company. So never regretted it. But we each work from our own home on projects for clients. And my social life consists mainly of sports, field hockey, but when that stopped, I was still in my pyjamas at my computer on some days in the evening. So then I decided I need to join a larger company. So I could also get a cup of tea for my colleagues or they get me a cup of tea. So I applied for a job at bol.com. And that was in 2007. 

 

So I started as a UX user experience business analysts where you describe functionalities that need to be added or improved to... in that period, only the website and later the app. So it's also nice on the cutting edge between business and IT. And always from the users point of view to make the application better. And I think that this driving force is also why I find the current job. So Product Manager for Team Experimentation, the most enjoyable job I ever had at bol.com. 

 

So with Team Experimentation, I enable others to learn what's the most added value is for their products. And I'm also interested because I can apply all the knowledge that I get during the studies in a long time I worked at Bol.com. 

 

So in the end, we don't only want to run A/B test on the website or in the app, but also, for instance, do experiments in the warehouse about picking routes. And then that maybe we get to that later. It's also not only about A/B tests. For me, it's all about learning and gain insights and become smarter.

 

Gavin Bryant  06:05

So thinking back to those early days at WebTastic, what are some of those learnings that you took from starting your own business that have carried through to help you in Experimentation?

Denise Visser  06:22

Yeah, especially regarding being Team Experimentation within a bigger company like bol.com, I guess currently, we do have 3000 employees. And I feel Team Experimentation is kind of a startup, or self-employed company within the bigger company. So all our colleagues are our customers, and we want to serve them as best as possible. So in a way, we are also looking for the right product market fits like a startup does 

 

Gavin Bryant  06:55

I think that's a really good description. And that's something that I've written about myself from my own personal experience, and my felt sense from establishing scaling experimentation program to it felt very much like a startup. And it went through the various stages of growth over time. And oftentimes, it felt like we only had enough funding and runway for 6 months at a time. And we had to keep continually proving ourselves, and delivering value for the business to get more funding to keep growing and scaling. So I think that's a really good description of what it feels like to start and scale an experimentation program. 

 

So what I'd like to discuss next is, I typically ask my guests, what's their Experimentation thesis? What's their philosophy? Or what's their guiding principles for Experimentation. But in this episode, we're going to take a little bit of a different tact. And Denise has previously articulated her 5 Experimentation, one-liners. So we're going to step through and discuss those experimentation, one-liners. So Denise, thinking about those 5 one-liners number one, what does success look like?

 

Denise Visser  08:19

Yeah, so first, have to say that I totally forgot about these five one-liners. But luckily, the internet never forgets. And you were able to find my presentation. And I still like, almost five to one-liners, because I think to convince people that experimentation is added value to your work, your story needs to be different for each type of person. So maybe for my son, being the risk averse one I should apply a different story than that for my daughter. 

 

So the first one is, what does success look like? 

 

And I guess here, we are all aiming for better and better achievements. But how do you know when good is good enough? So what are the numbers of your performance today, but also last week? Last month, last year? And can you explain the differences between those numbers? And what's your plan ahead? And when do you know when you stand out from the rest? Because then you might even decide to start.... To stop working on this product or this service? And focus on other tasks in the company? Or do you need to stay ahead but what is needed to achieve that? Or maybe the other side could be that maybe you are already satisfied with your performance. But when you start comparing to others, you realize that there is still room for growth. 

 

So the question what does your success look like? 

 

It's about setting a goal but also being aware of your progress towards that goal. So are you already almost there? Or is there still a lot to gain. And I guess it's simple to us to take a look at the numbers every day, but a good understanding of the numbers, and why they fluctuate, this is definitely next level. And understand how these numbers reflect your success. And success is when the outcome of your product meets the users’ needs. So understand how you can measure this, and how you can have impact on those numbers. And of course, experimentation plays an important role, and to learn what the actual effect is of your efforts, and your input leads to output. But understanding what the actual outcome is bring.... Yeah, that's important. And what I also do like about this approach, so especially when you have a wise guy who says, just implement this, because it will be successful, then you can also ask the question, but aren't you curious about how successful it will be. And then it's also an easy way to convince that Wise guy, that's an experiment has added value, because he cannot only state that it was a success, but he can also actually link numbers to it. So that makes it even better.

 

Gavin Bryant  11:26

That's a good point that he can then understand the magnitude of his success, rather than success being a binary outcome. So just talking about success, more broadly for a moment, Experimentation Teams can often struggle to articulate success of their programs. And there's many different ways that success can be measured, it can be to your point customer experience, it can be revenue, it might be budget saved. From your experience, what advice would you have for teams trying to measure and articulate the success of their programs?

 

Denise Visser  12:10

So I guess we didn't crack the nut yet. But what we do is that we communicate about the number of experiments that ran, in the previous month, we report per month, and also the number of involved teams, we also have a calculation in place about the additional revenue. But that's just an indication, but it turns out the business people think it's a factor that this about definitely will be added to the profit or to the revenue. So, I am looking for a different solution. But up until then, I think those numbers are easy to communicate, and then people want to exchange their achievements. And if there is in some way, a possibility to rank, then it also helps. But for me, as an experimenter, there is maybe no better experiments, or any additional revenue might not be the best metric. So, I'm still thinking about what can we do to reflect even more the quality of the experiments and the more long term impact of it. So it's not just about hit and run have additional value, but also that you want to learn from your customer and know how you can serve them better.

Gavin Bryant  14:12

So number two, is thinking about smarter teams, the smarter team gets, the more effective a team gets. Let's talk about that one for a moment.

 

Denise Visser  14:25

Yeah, so I guess it's very easy to put additional tasks to your list just because it's might have a positive impact on the outcome of your team. But I guess in every company, time is scarce, so when time is up, some tasks fall off the list. And to make sure that you have at least done the tasks with the biggest impact you need to know about what causes change in the behavior of the customer. And you have to experiment to figure out what works. So you might recognize also in your company that the list of tasks and ideas is always longer than the available time. And at most companies, there are a lot of ideas, initiatives, no-brainer suggestions, demands, or you name it everybody wants something of a specific IT team.... Of all the IT teams together. And companies do several methods to prioritize these tasks. You're not alone with the person with the biggest mouth, get the most out of it. But what if we could do this prioritization, not on gut feeling of the hippos or the zebras, and it would make our life much easier if we can bring proof to the table. So validated by experimenting makes your team smarter to learn in which tasks they should put everything in. And of course, you learn in small steps. So also break down your ideas up in small pieces, go for the Minimal Viable Product and then check whether there is a product market fit or whether it was an improvement of your current solution.

 

Gavin Bryant  16:10

Good point. Just discussing that a little more broadly. So innovation teams, experimentation teams, that the demand for their services is always very high. And you mentioned that there's generally too much work to be able to address, too many projects, initiatives, concepts, you find that your team, you need to be absolutely ruthless on prioritization, given that you only have a dedicated number of resources to serve the business. Is that fair to say that the team needs to be really sharp on what they take on and what they don't?

 

Denise Visser  16:58

Yes. So I guess, first of all, it's not only for Team Experimentation, I guess it's for all teams within our organization. And I guess in all organizations, it's not just a game of you ask and we deliver, of course, you need to have a good understanding, but what the actual outcome over change or a new functionality might be, and that is not only the responsibility of a manager or a project lead, I guess, it definitely helps if everybody within the team has a good understanding of how you can make impacts to your users. So if there is a request, then I think it's a good idea that's also discussed by the team what the priority should be.

 

Gavin Bryant  18:03

Good point. So thinking about number three, now of 5 one-liners, without experimentation, you're testing without knowing. And you talked to this a little bit earlier about the wise man, that he thinks his initiative may be successful, but he doesn't know how successful or if it will be successful. So maybe the wise man is not so wise, he's just guessing. So what are your thoughts about testing without knowing?

 

Denise Visser  18:36

So definitely, in the same reasoning of the previous one liner, but this one I like, specifically, because often, it's only celebrated, that the release went live. And I guess that we also have, of course, people celebrating that part, but also put more effort in learning about the actual impacts that's the new functionality or the adjusted functionality has on the end user. Because then you start knowing and then you start learning. 

 

So, keep track of all the releases that are done in your company and keep track of what the actual impact was on the user behavior, and therefore the success metrics of the company. So then you can learn from the past which hypotheses turned out to be true and which not. So it is said, I don't know I don't have a source for this fact. But more than half of what is built in a company has zero or negative effects. So I guess it's especially maybe also for starting experimentation teams, find out learn what this number is for your company, because this is room for growth. And maybe it's the end of the scarcity of resources. But I can imagine that we can come up with plenty of ideas that we still keep everybody busy.

 

Gavin Bryant  20:25

I believe that statistics that you referred to was reported by Microsoft from memory.... They reported that 30% of changes had neutral or no benefit to users, and another 30% of those had a negative impact. So effectively, more than 60% of changes had no benefit to users at all.

 

Denise Visser  20:53

And then I think there's also a number available within the Microsoft environment that's only 10 to 20 of their experiments is as a significant outcome. So yeah, that's in the same range maybe.

 

Gavin Bryant  21:10

So just touching on a practical element for a moment, you mentioned that it's really important to not only be testing our current hypotheses, but also reflecting and reviewing on past tests and past hypotheses, do you have a really good way of storing all that information and making that available across the business?

 

Denise Visser  21:37

So as Team Experimentation, we also are the owner of the product experimentation, and that is also an experimentation platform, which has several goals, but one of it is administration. So we store all the experiments. So, you cannot configure an A/B test or you cannot kick off an A/B test if it's not stored in our database. So that is an easy start. But then we also ask you for what were your learnings, for instance, or add some images to it that it is easy to recognize for future use. What the experiment was about and of course, we also save the hypotheses. And so as much as insights as possible, related to the experiment, this is stored in our experimentation platform.

 

Gavin Bryant  22:39

Excellent. So moving along to one-liner number four, and I love this one; learning before earning.

 

Denise Visser  22:50

Yes, so it's also related, I guess, to what I said about the number of Microsoft and how many experiments are successful in a way have a significant impacts, because every experiment is successful, because you can learn something from it. But what I wanted to say is for don't be too harsh on yourself, that not everything you release is successful. And so also professional football players don't enter Camp Nui or Enfield out of the blue, they practice lots before they reach that level, and there are also a lot of one-liners available to also make this point. So, there's nothing like a failed experiments, only experiment with unexpected outcomes. And every time you do a new experiment, you learn more, you cannot learn less. And so never stop testing hypotheses after one run. Please dig deeper and come up with another treatment. 

 

Gavin Bryant  23:58

Just a couple of things …

 

Denise Visser  23:59

Keep on going.

 

Gavin Bryant  24:00

Yeah, that's a good point. Think of the hypotheses like an onion that you just keep peeling off all the layers of the onion until your learning objectives have been met. And I just have another question, you talked about unexpected outcomes. How often do you see unexpected outcomes of experiments that you're performing, whether that's a second order effect, or a third order effect that you didn't think about before performing the experiment?

 

Denise Visser  24:38

To be honest, I guess most unexpected outcomes is that the data you have after the first one is not complete enough that of course we want to motivate people to think everything through upfront, but then still you encounter stuff that you are curious for. But then you find out I didn't measure. So as Team Experimentation, like I said, we provide a tool, a platform. So that is an IT solution. But we also are a group of consultants within the company. So if you want to start experimenting, you can ask us for advice, and we can think along, and then we try to prevent the scenarios that you are surprised by it's not measured properly, for instance. And I guess, this, what you're stating is maybe not happening a lot at bol.com. But it could also be because we are, we do have a kind of an overview. So we are aiming for increasing conversion, and we might not even look in the increase of customer service calls, for instance. So what we do as a Team Experimentation is that within our platform, if you run your experiments after the running time is over, then there is an automated result page. And of course, we give you insight in about whether there is a significant result on your chosen KPI but we will also provide you with some health metrics. And that is about customer cases. So how many customer cases did the control had and how many did the treatment have but also returns cancellations, we also want to add the performance also the loading time for the users to it's.... Just to make our colleagues smarter so and also to widen their view that they are not only should be interested in the conversion, for instance, but also in maybe possible downsides of their experiments.

 

Gavin Bryant  26:58

Yeah, that's a really good point to consider the upstream and downstream impacts of the experiment. Just quickly, you touched on that. Sometimes there's the potential that teams perform experiments and the data and the insights may not be so reliable due to the measurement or the design of the experiment. Is that the biggest challenge that you face day to day?

 

Denise Visser  27:23

Yeah, that could be the biggest guess. Because it turns out that it's hard to understand the data. So, you have to have a good understanding of how the website works and how its measured and how to interpret all the data. And then again, as Team Experimentation we... So our objective is to make it ridiculously easy for our colleagues. And then at the beginning it was to do experiments, but now it's.... Where it's more broad, and to work evidence based. 

 

And in the back of our head, we do have three sub goals. One is spread the virus, the experimentation virus, so make people enthusiastic about that experimentation is absolutely valuable for each team within bol.com, master the data. So that's why I started this, it is also part of our goals, because we want to understand all the numbers and all the reasoning behind it, just also to explain it to colleagues, especially in a fast growing company, it's hard to keep up with all the complex materials. So we want to make it easy to understand for everybody. So we also feel the responsibility for the data within bol.com. It is different products and a different team [phonetic 29:17]. But sometimes we are thinking about adopting it, just to make it more and more our own and to be complete. So the third sub goal for us is create experimentation heaven that’s more about usability of the platform, and the easiness of setting up an experiments.

 

Gavin Bryant  29:42

I love those three objectives, they are very real and they're very easy to bring to life in the business. It's our final or 5th one-liner; work smarter, not harder.

 

Denise Visser  30:01

Yeah, so that's my favorite one. And I wish it's for everyone the joy of knowing what the actual impact of your hard work is. And that you can tell to your friends or to your mother, not only what you do, but also what the outcome is. That's great. And, of course, there are a lot of ways to become smarter, but experimentation, and then in a broader sense, gaining insights, makes you smarter, and makes it easier also to work smarter. So, what we currently are doing as Team Experimentation our A/B tests are still our core business. But we already have the database in place, and we want to also store other insights. So maybe interviews with customers, or UX research, market research, maybe simulations we have done, process mining results, whatever we add, now, it's kept in email boxes, or directories, or whatever. And especially for new people, it's hard to retrieve all this information. So we all want to get it in our database. And maybe we have to rename it. And it's not an experimentation platform anymore, but the research library or the Learning Center, or whatever. But all... Yeah, to make our colleagues smarter, so that they can decide what the best thing to do is.

 

Gavin Bryant  31:44

Yeah, I think that's a really good initiative. That's one of the challenges that I've experienced in my work as well is that there's so many different types of customer research that are being performed across the business, to your point; user research, different forms of qualitative research, quantitative research, experimentation, and it can be difficult to aggregate all that information into one central repository. So yeah, the notion of a centralized customer research library, it makes a lot of sense.

 

Denise Visser  32:23

It's not about good or better. So an A/B test is not better than a market research. It depends on the situation and the maturity of your product and the availability of the data, of course, so please investigate and become smarter is the broader news or the.... I don't know the right word for it. So it is about experimentation, but it is about learning.

 

Gavin Bryant  32:55

Hmm, yeah, I agree that all of those forms of customer research that are performed need to come together to be able to make a reasoned assessment of the business opportunity or business problem that one is not more superior than the other. It's really bringing them all together to provide a 360 degree holistic representation of the customer.

 

Denise Visser  33:22

Yeah, exactly. So that's also our relationship with the different teams when at bol.com. We are experts regarding experimentation and do running A/B test. But I expect that they are the experts of their products. If you're responsible for payment methods, for instance, or for product descriptions, then I expect that you know everything about it, and that you did the research and that you can come up with the right hypotheses. And of course, we can challenge you because we know what the good hypothesis is like. But the actual content is the responsibility of the product teams itself.

 

Gavin Bryant  34:07

Yeah, I agree. So let's jump forward a little bit. And let's talk about the journey of establishing experimentation at Bol.com, from very humble beginnings to highly successful and award winning program. What did experimentation look like in the early days?

 

Denise Visser  34:29

Yeah, I guess there was no culture of experimentation. So as Bol.com we are proud to be an innovative company and we were definitely guilty of putting something live and not looking back and immediately focus on the next new feature. And I think and not to blame the company because I already worked there for 14 years and I'm a bit in love with bol.com, so no hard feelings, but... So I guess it It was mostly because of enthusiasm and impatience and seeing all those new possibility that we decided, okay, let's done, let's move to the next. And to be honest, it also brought us a lot of success, because we were often apparently right. So we've managed to grow every year. But if we had been more critical back then we might have managed to grow two times faster, we will never know.

 

Gavin Bryant  35:30

Hmm, that's a great point that those early days, maybe more luck was at play, rather than quality decision making.

 

Denise Visser  35:42

Yeah, you never know.

 

Gavin Bryant  35:45

So moving along.... So initially, there was no culture of experimentation that there was some loose innovation happening within the business that how did the experimentation program come about?

 

Denise Visser  36:01

So it started with a group of enthusiasts, I guess, the CRO profile already existed. So conversion rate optimization, but we didn't have any CRO specialist. Again, as bol.com be expected from everybody that they were optimizing conversion. So why is it a specific role? But we started with a group of enthusiasts, so some business analysts from the webshop, and the app, we had some web analysts in our group and also now and then a developer joined the club. And we made use of the scarce experimentations options we have. So there was no official tooling at bol.com we never bought a solution. But dependent on whether the project lead thought experimentation is important, the functionality was built, and I guess, even an intern built the bucketing system back then, that was used by all other teams. 

 

So sometimes the functionality of testing was implemented. But it's hardly ever used, because already the focus has shifted to the next topic. But we sat together I said, and we came up with okay, what can we do and we decided that we set ourselves a goal to run 100 experiments in 2018. And no matter how successful those experiments were, but just to learn, and it turned out, it was hard to organize 100 experiments in a year. And the fun fact is now that we have more than 80 experiments runs for a month, and still growing. So what was hard back then, is already way easier right now, luckily. 

 

But with the amount of tests we did in 2018, we were able to bring in an additional revenue of 100 million euros, which is a lot of money. And of course, it was about low hanging fruit, because we only put first iterations live and never tried to improve it. But still, it's a lot of money. So this opened the eyes of the board and the managers. Because what if we could achieve this with only small adjustments in the front end, just change some texts, different order of elements of product pages, for instance? What if we could also do this in the back ends, and all system logic under the hood, so that was the beginning of Team Experimentation in 2019.

 

Gavin Bryant  38:47

It's an interesting and unique start to a program that the program started off very small, very local. And it was characterized by interested, passionate experimentation champions, and the program has grown out of that very small beginning based on some very good quick wins that were established early to demonstrate significant revenue uplift. Do you have a view on this...? There's lots of thinking that experimentation needs to happen top down from executive leaders, and it should be a top down strategic mandate. In your experience, you have established somewhat of a grassroots organic movement. Do you have a perspective on that?

 

Denise Visser  39:44

Yeah, of course, it helps if it's a top down decision that you do monitor rollouts, for instance, and that you want to be aware of what's happening with every release. But what I'm hoping for is that it will also start but because of curiosity, and that you're interested in what actually is happening with the product you create and try to improve. So I guess you need both top down or bottom up. And I guess the main thing is just do it. So like the group of entrepreneurs I just described within bol.com. The only plan we made was, let's do 100 experiments. We didn't ask for permission, or whatever. And we just thought it, and got result out of it. 

 

Gavin Bryant  40:51

So thinking about some of those early hurdles, that you mentioned that the objective was initially to just try and do 100 experiments. And now you're nearly doing 100 per month, what's changed over that time to make experimentation easier?

 

Denise Visser  41:11

So to be honest, the scope for the small group, the small group of answers was also smaller, was only about front end. And now we can cover the whole area but still, so the whole area is we do have 200 IT teams within bol.com. And our goal is to have them experimenting all but the IT landscape of bol.com is rather complicated. 

 

So a lot of services and a lot of those teams are entangled. Which means that there are a lot of dependencies between teams. And I am blessed with only smart people in my team who also have a good knowledge of this complex IT landscape. Also, because they are already around for longer periods. So we can guide a lot of IT teams. We grow fast as bol.com during COVID, 1400 new employees started. So that makes it extra useful for Team Experimentations to make solutions that can be used by anybody. And to make all the teams as autonomous as possible. 

 

There is... I think, it's too technical to explain how it works right now. But I'll share the link with you, then you can put it in the notes, we created a solution that we call a beacon ID, if a back end server [phonetic] want to run an experiment, but they are also interested in the actual impact on the user in the web measurements, normally, the front end team needs to work along and have to implement the experiment themselves as well just for measuring stuff. But what we introduce is the beacon ID is that with every request from the front end to the backends, the beacon ID sends along and that is the linking pin to what's happening in the backend, and what's happening in the front end. And the beacon ID is there already. So there is no need to adjust that for every experiments. So I guess that's a nice example of what we achieved as Team Experimentation to make it easier for everybody within the company to actually start an experiment.

 

Gavin Bryant  43:43

So thinking about advice for businesses who are looking to make a start with experimentation, what would your key pieces of advice be?

 

Denise Visser  43:54

Like I said, just start, I guess that also didn't prove whether it's valid for your company or not. And don't give up after the first experiments. But also prepare your audience that the first experiment might not be successful. And of course, that might be a bit harder to sell. But I guess, if they are already aware of what you actually want to achieve, and that is not only winners, but you want to achieve learnings, then it's easier to continue after first failing experiments. And of course, if you are starting as a company with experimenting, I'm convinced that there is low-hanging fruit, and that there are no brainers and maybe if you're more mature then you don't even test no brainers and of course, there may be a no brainer. There's a no brainer, nice discussion in itself. But yeah, pick those. Look around and see what others are doing and try for yourself if it also works for your company.

 

Gavin Bryant  45:19

That's a good point about no brainers. That's a whole another discussion, isn't it that? We would tend to believe there's nothing as a no brainer. Just one closing question around the experimentation [email protected]. So establishing this ground up capability that started small with a curious and passionate team. How have you? Or what strategies have you found effective to build experimentation capability more broadly across the business? Is that partnering closely with the teams, workshops, document information? What are some of those things that have worked well?

 

Denise Visser  46:11

So the most important thing maybe is tell everybody what you're doing. And I had to learn it by myself as well. But luckily, one of the group of enthusiasts was very good in sending emails, telling everybody about successes, maybe even before we reached the final outcome of an experiment that helped a lot. And now we are teaming up with a lot of different teams within bol.com. So for instance, we do have data coaches, who explain and then not only about what measurements, but all types of data we have within the company, how should it be used? And where can you find everything. And if you understand the data, then experimentation is a nice next step to make use of it. 

 

It's recently decided that bol.com will become a product led company. So I think it's no rocket science. And it's nothing new in Silicon Valley. But for us, it might be new. But there was also a team involved in changing the way of working and I guess also, as Team Experimentation, we try to change the culture within the company that you are looking for proof and that you want to validate. And now we have a nice opportunity to work along with those people want who to make us more product lighter than we currently are. 

And what we also do there and more in communicating our successes. So we do have workplace as a communication platform within the company. We decided for ourselves that as Team Experimentation, we want to publish at least 25 messages per quarter, just to tell everybody what we are doing, what is going well, what can be improved or whatever. But keep it top of mind. And that helps also a lot and what I mentioned earlier, if you need to start a Team Experimentation in your company, I guess it's very helpful if you have members in your team who are already there for more than 5 years, depending on your company. But that definitely helps because then you have a good network and you have a good understanding of the challenges. So if you are able to create your own team, please flick pick a senior.

 

Gavin Bryant  49:01

I think it's a really good point you make there that communication is key and critical to the success of not only experimentation program, but also to increasing organizational learning. And my opinion and perspective is the role of experimentation is 50% performing experiments, but also 50% of time should be spent with communications, influencing and stakeholder engagement and management. 

 

Denise Visser  49:31

Yeah, perfectly. 

 

Gavin Bryant  49:33

So let's just work into our quick three closing questions now to wrap up this show. So first one signature question, a series of experiments that have changed business thinking.

 

Denise Visser  49:50

So last year, we did an experiment it was called Optimal Delivery Day. And it was not only about increasing conversion, but it was about what's the Optimal Delivery Day. So at bol.com, the default delivery day is tomorrow. That is standard. But we have learned from our customers, that is fast delivery is not always needed. But because we put it there as a default, it was harder for the customer to change it than to keep it the same. But especially during busy times, like Black Friday or Christmas holidays, the workload in our warehouse is big. So with those experiments regarding the optimal delivery, we were looking for, how can we change the default delivery day and then postpone it with one day, for instance, but keeping the level of conversion at the same level? 

 

So that was very interesting. And especially because a lot of different departments within the company were involved. So not only the selling part, but also the logistics, because they had... There was a possibility of a big game for them. So it was very nice to have this conversation as well.

  

Gavin Bryant  51:18

So the key learning to come from that series of experiments were that customers were willing to accept a slower delivery, and there weren't observed impacts to conversion. 

 

Denise Visser  51:32

True.

 

Gavin Bryant  51:33

Okay, excellent. So thinking about some resources that you would recommend to our listeners and our audience. What would some of those key resources be books, any newsletters, or blogs that you'd recommend?

 

Denise Visser  51:52

Yeah, so I thought maybe I should promote some Dutch initiative experimenters. So first of all, I'm a big fan of conversion hotel. It's a three-day conference on Texel in the Netherlands in November, if you're able to join, you won't regret it. And then you also you, you already had Ruben deBoer in your show. And so he is also providing experimentation courses. But I'm also a big fan of Peep Laja. He's not from the Netherlands, and I also want to mention, Guido Jansen, he is also making a podcast about experimentation. So he started already a couple of years ago, and I can imagine that you also heard of him. And Kevin Anderson, and Tom van den Berg they have both their own newsletter about experimentation. So I also love to read those as well.

 

Gavin Bryant  53:03

And final question, if listeners want to reach out to you and get in contact, what's the best way to reach you?

 

Denise Visser  53:10

Yeah, so stop by Bol.com headquarters in Utrecht in the Netherlands, but if you're not able to, then of course you can reach out via LinkedIn. So, on LinkedIn, I'm happy to connect.

 

Gavin Bryant  53:30

Thanks so much for your time today, Denise. Great chatting to you.

 

Denise Visser  53:34

Yes, thanks for having me again. Enjoy the rest of the episodes.