Hello, and welcome to The Good Practice Podcast from Emerald Works, a weekly show about work performance and learning. I'm Ross Dickie and this week marks the return of The Good Practice Book Club. Back in December, we kicked things off with a review of Malcolm Gladwell's ""Talking To Strangers,"" and this week we're casting a critical eye over Stefan H. Thomke's ""Experimentation Works."" I'm joined as usual by Owen Ferguson. Hello Owen
Ross Garner. Hello Ross.
Hello Ross, boo
Nicola Boyle. Hello, Nicola
Boo. I misread the room. The audience are not on my side here.
So let's get things started by just talking about what the book was about. Ross, do you want to give us a brief summary
Sure. So it is written by a Harvard Business School professor, Stefan Thomke. I think it's how you pronounce his last name
That's why I went for it. I went for it. I tried to be confident.
Yeah, let's go Thomke
I have no idea.
So, his argument is basically that the decisions that we make in a business are systematically flawed. And so we implement ideas and strategy based on our intuition and gut feeling, and often the opinions of the highest paid person in the room
So Thomke, his suggestion, is that a better, more humble approach is for us all to adopt an experimentation mindset. So as far as possible, every idea should go through an AB test, where you measure what's currently happening. That's what Thomke calls ""the champion"" against the new idea, which is the challenger
So as an example, you might change the color of a button on your website, but only show that change to a random subset of users and then you compare the results for the two groups. Often the original button will perform better or the same as the challenger, in which case you keep the situation as it was. But if the challenger performs better on whatever metrics you decided, then you'd roll out that change more widely
If the AB test fails and your new idea was not a better one, it's not a failure because you've learned something new. As long as you have done a rigorous test, that was a good idea to start with, and then you can use that information to inform future ideas and you keep doing that as an incremental way of optimizing your business
And so the title of the book ""Experimentation Works"" is pun on experimentation works, hurrah, we should all experiment and also your business should become an experimentation works as in a place where experiments are created. How's that for a summary
I thought that was very good.
Yeah. Thank you very.
Well done. Well done.
It's fair to say, I think, there's two halves to the book. The first is kind of like making the case for doing more experiments and business. Then the second half concentrates a little bit more on how do you make that change
And possibly talking you out of doing so. But we can get into that a little bit later on maybe.
Nicola, what did you think of the book. Did you enjoy it, for starters
Yes, I did enjoy it. I thought it was interesting on how to run concurrent experiments in large multinational companies. I particularly like the chapter on insight and experimentation organization. So it takes quite a deep dive into booking.com. And I felt that that chapter in particular was quite interesting because it just gave you some insight into how they run those kinds of experiments every year; how everyone in the organization, no matter who you are, is subject to- anyone's ideas are subject to an experiment. So, that was the chapter I probably got the most out of. And I thought it was most interesting, but it was quite thorough, especially the first half
Oh, see. So you picked up on there, the opposite of what I was thinking, reading that chapter. So you described that chapter as how to run experiments in a large, multinational organization. What I thought was that, this feels like it only applies to a large multinational, organization and has put me off the notion of experiments because the scale that Booking run experiments, seemed so unbelievably difficult, that the disconnect from, what could I do personally, to then everyone in this organization is doing loads of experiments all the time just seemed too vast. And I also got slightly annoyed at the- Thomke would argue that booking.com and other large online businesses would run lots of experiments at the same time, but often always talk about the example I used at the start, where you would change the color of a button. And I kind of wanted to know what some of the other experiments were, because that just seemed like the most, like the easiest example to explain. But is it the only thing they're doing is just tweaking the contrast of colors all the time
Yeah. I thought similarly, when the example is given about booking.com CEO and how he was also subject to, his ideas were subject to experiment. So he said that he would like to change the local. So they said that they would test that, which to me seems like common sense. They would do that for such an important part of their brand identity. So I wasn't totally convinced by that argument either
No, I was convinced for that one. I'm just going to disagree with you because that was one of the most powerful messages that the CEO could come up with an idea, and the instant reaction is not ""Let's implement this idea,"" it's ""Let's test it."" That was obviously like one of the biggest hurdles to creating an experimentation works was, you need the highest paid person to be humble enough to accept that book
So I was torn on that example because I think it does send a powerful message around, not just being the highest paid person, but then if I look at some of the previous chapters and it's talking about how well could you run that experiment, right? So you want to change the logo. Now, if you're going to change a logo, you're changing it everywhere. And there's usually some kind of brand relaunch or brand activity to get the logo out there. Whereas if you're only just tweaking it in situ, you're only showing a subset of users the logo on the website, then they're not getting all the additional contextual information around that. So, to what extent can you test launching a new logo in an experiment in a way that you can control for all the other factors that would be involved in that? I just thought that would be a really tough experiment to run
Because by the time that you're at the scale of booking.com the people who are coming to your site already have some brand recognition. There's probably not a lot of new people who are like, ""Oh, booking.com, what's that?"" I'll go, ""Oh, I like that logo, I'm going to buy something here"" versus, ""That logo looks slightly different, well, I'm still here to book a holiday."
Yeah. Although I think with those things you're being so facetious because firstly, it's just no a conscious thing or I like that logo, I'm going to here. It's like, yeah
They're looking for... particularly if you have millions of users, which is another problem, if you have millions and millions of users, then you can look for very, very small incremental gains on what you had before. But if you have a thousand users, which I think they describe as the minimum you should have for doing AB test, a thousand users a month, you're looking for larger effects
I think, does it not... Was it a thousand daily users, a thousand monthly users? It was a reasonably high bar that suggested that this is more for external stuff than internal stuff, depending on the size of the organization, but not many properties get a thousand daily users
I think it was a thousand a month. Cause I think I say to the same thing in my dissertation. But check the show notes and there might be a correction there.
But I don't think that necessarily argues against conducting experiments, it just takes you longer to get results. So you can't run as many concurrent- your experiments at the same time, you need to really sort of focus in on what are the highest value? And this is where the sort of, I think that the point that he's trying to make is, you should be experimenting with everything. What if you can't? What if that's just not possible? How do you then make decisions to optimize what you're doing? And that is where expertise, judgment, intuition starts to come in
Yeah, I'm sounding very negative. I actually came through the first half of the book, I think this is why I took against it in the second half and the first half I was so excited. I was really keen on experimenting with absolutely everything. And it just seemed so difficult in the second part there, it seems like you don't know where to start. But he does acknowledge that he has that five part kind of maturity scale of you're only doing one or two experiments to start with. And then as enthusiasm for this spreads, you can start to build on that. But then that process is going to take years. It's not going to happen overnight
Yeah. I mean, I just, I generally, I mean, this is one of the things that you said about the book, Ross, as well as with a lot of books of this sort, the examples are cited are large companies, kind of all the same companies. It's like booking.com, Google, Amazon, although to be fair to most books like this don't cover the team New Zealand America's cup sailing team. But I think the sort of- I don't think he was explicitly trying to do this, but almost seemed like he was trying to say that the reason all of these companies are successful is because they run experiments. And it's sort of almost seems like an oversimplification of why these companies who are all very well known have achieved the success that they have. And yeah, I think- I don't think he's explicitly say supposedly trying to make that argument, but does sort of come across as like look all these things that neatly fit into my argument
Yeah. I agree with you Ross. And one thing I also thought was all these companies make such a huge amount of profit, would they've been as willing to carry out so many experiments earlier on in their company lifetime considering they have this buffer of so much profit already
Yeah. Because one of the examples it gives is like the domination of Google over internet search and refers back to when it was sort of more competitive. And the reason that Google has stripped Yahoo is because of their experimentation culture. But Google was still a relatively large organization at that point. So it's still sort of trying to figure out how... I guess my next question would be, how do you see Thomke's work applying to smaller organizations? Or even more specifically to learning and development
All right, I'm going to ignore your question and go back..
You thought you were hosting. In fact...
My response is not unrelated, but it is..
I'll be the judge of that.
I wondered how far it was a survivorship bias going on. Cause I suspect a lot of online, now Owen has put an expression that's difficult to read. I don't know if he agrees or not
I agree with you, yeah
Okay. So you look at the largest, most successful companies in the world and talk about how the experiment as a possible causal mechanism for that. But you ignore all the other online businesses that experiment all the time that haven't experienced the same success
Yeah. I think, I mean, this was an argument in the book- and I should probably say, because there's a danger now that we're sort of, we're starting to nitpick here. The book is excellent. It's a really a clear explanation of why experimentation is an important tool in your decision, making a toolkit, if you will
Yeah, particularly the first half of the book
Particularly the first half from making the case.
But I think there's aspects in the second half that you can learn from and adapt to your own context. I think you can say that an experimental mindset has contributed to the continued growth of those organizations. But you know, as Nicholas mentioned, there is something inherent in the business models of those companies that came from serendipity. So you have to get lucky first and then experimentation can make you more successful. You know, I don't think, I mean, Ross documents survivorship bias by definition itself, all the companies that are around today haven't fallen. We're looking at a set that have overcome survivorship bias in some way, shape or form. But once a company reaches a certain level of maturity and probably has got the scope to think a bit farther into the future, so they've kind of got of that very early stage of initial wild bets, experimentation can be a very useful way of helping you decide how to progress
Yeah. He does make a distinction as well between controlled or disciplined experimentation and experimentation sort of foreign so and sake. So another example he talks about earlier in the book is, I can't remember the guy's name, but formerly- basically the guy behind the Apple store was recruited by JC Penney to try and replicate the success. They'd had in Apple, which I didn't appreciate. It was apparently the most profitable retail space per square meter or whatever, of any organization. And basically he tried to take his- you can argue about whether or not this is actually an experiment or whether he was just replicating what he'd done at Apple for JC penny, but sort of overnight, or very quickly got rid of checkouts and completely changed the layout of JC Penney stores. And it was a disaster for them. Because that, I mean, I think he would probably argue that wasn't a true or a sort of control disciplined experiment. So he's not necessarily saying that experimentation is always a guarantee of success, but it's sort of, it's talking about like the right type of experiments, I think and the way that you approach them
Yeah. I mean, it's very much talking about the scientific method. That you develop a hypothesis and find a way of testing that hypothesis, usually with some kind of control group. So this is the thing with JC Penney is that he just to say, we're going to make all these changes, he didn't try all them out in a number of stores to find out how people reacted to them. Or at least that's the impression that I got from the book. The reality might have been a bit more messy than that, but it certainly didn't seem to be kind of a structured, controlled trial that they then use to make decisions about whether to push on further or not
One of the things I found really interesting was the Facebook contagion example from the book. The ethics of experimentation by other people like finding out that they were split into an A and B group and then saw different stuff and that the outcome measured. Cause there was a, I remember this being in the news, the story was that users on Facebook, a small subset of them were showed either positive or negative stories. And then the impact on the sentiment of their subsequent posts was measured. So these were all real stories. They weren't made up, it was actual positive or negative stuff from their timeline. But it was filtered in such a way to bias it towards one or the other. People were really annoyed that their emotions were being manipulated. And it actually found much of an effect whatsoever
But Thomke's point was that this is, it's kind of a weird reaction because it assumes that the thing that already exists, the default is the way that it should be. Were they exploring whether what you're experiencing might be sub optimum. So for the JC Penney thing, it's like, you shouldn't just assume that the way that the stores are laid out is the best way, but you don't want to go full throttle to changing it until you've got some evidence that doing so is a good idea. And it would be a success to do that experiment and find out that it's not a good idea before you rolled it out
Yeah. I mean, we've spoken about that before is not making the assumption that the intervention that you're going to put in place is going to have a positive or even a neutral outcome, that there's every chance that it will have a negative effect. And so, if you are rolling something out without having trialed it or tested it in some way, shape or form first, then you're potentially causing harm
And one of the interesting little snippets from the book, Microsoft finds that one third of its experiments prove effective, one third have neutral results, and one third have negative results. That seems remarkably neat and tidy to me. But on the flip side of that, Google's experiments fail at 96% of the time. Now, they're running a lot of experiments. So you would expect- I wonder to what extent, the more experiments you run, the more likely it is that you're going to have failed experiments, but more likely that you'll uncover the ones that will have the highest value to the organization. But certainly having that mindset of, the new thing that I'm doing might be detrimental, should act as it should act to give you some kind of caution about just rolling ou
That kind of sums up Google's approach to their more innovative ideas as well, though. Because one of the problems I found with the book, and Thomke addresses it a few times throughout, is that if you're only going to focus on incremental experiments, then you run the risk of ignoring the kind of big board ideas that may be truly transformative rather than a marginal gain, like something that's really going to be transformative. And so, although Google does run incremental experiments, it also buys up a load of companies expecting some small percentage of them to turn into a huge success and being willing to accept that the risk. That a lot of them will go nowhere, but you need a lot of capital behind you to be able to do that
Yes, I don't think that's- while that's interesting, it's not something that I can see us applying in our day-to-day working lives
I've got five companies I want you to buy, Owen. Maybe even one of them turns into your billion dollar proposition
So talking about day-to-day working lives, I'm going to return to the question that I asked that Ross chose not to answer earlier, which is possibly because there's not an easy response, but how do you see this applying to L&D
So, I think we work in an area where experimentation is not the norm. So while we are not necessarily going to turn our organizations into an experimentation works, we could experiment with what we're doing. Cause there's all sorts of things that we do that we don't know if they work at all. So there was a nice idea in the book which was to have an experimentation checklist for all employees and I think that this would work quite well in L&D
So do you have a hypothesis was the first question. So what do you think is happening? Can it be tested? That's if- not everything can be tested, but can it be? Do you have enough data to test it? And then once you've kind of answered these questions, I think you're in a pretty good place to do whatever intervention you had in mind. And even having thought about how you would measure the impact of it at the start will help you assess whether it was successful rather than going, I'm just going to do this thing because it's a good idea
Yeah. I mean, we have spoken about this before quite a bit, but there is definitely scope for applying a more scientific approach to what we're doing. Certainly in terms of trialing things. If we're replacing programs, to run them in tandem for a period and measuring the longterm impact, but one of the things that he mentions in the book is a learning mindset is basically an experimental mindset. It's doing your work in a way that you can learn from the outcomes and apply it in our kind of fractious cycle to what you're doing. And so I think there's two aspects for- that L&D can look at. One is, how do we cultivate this kind of learning culture? Is that something that the organization wants to do and how can we support doing that
But the second thing is how do we apply this to our own working practice? And there's no need to do absolutely everything overnight. And I would say that it's probably easier to run experiments in an online setting than it is to do it in other settings, where it's by no means impossible, I just think it's easier to do it in that setting. Pick something and try and apply some of the principles. And from that, you'll learn how to conduct good experiments that give you information and data that you can then use and apply to other interventions that you're running
So we did an experiment recently where, and there's a blog on this coming, we have been running various compliance courses for Emerald group, which is a company that owns Emerald Works and rolling them out. And it is a total pain in the arse chasing full cup to complete these things. And so, one of the things that we often hear from clients is, we should get the CEO on camera or delivering some message in some way to support this thing cause then people will just do it. They'll do it cause they've been told it's important by someone in authority. So we did an AB test. So myself, the facilities team who were behind the compliance training, and then internal comma got together, we AB tested the rollout of our display screen equipment assessment e-learning. So 50% of employees got a message from the CEO, 50% got a message from workplace and facilities
It maybe made a very small difference on uptake for the CEO group, but not a statistically significant uptake. So was that a field experiment? I don't think so, because results were a little ambiguous, but what we found was that if you think that getting the CEO to tell people to do something is going to be the answer to your problem, in this context, it wasn't. It didn't make much of a difference at all. So we're not going to have every piece of compliance come from the CEO's office in the future. And it wasn't that difficult to do. We just needed to speak to the people behind the project. And then also the internal comms to market
I mean, you can start like, I think as Owen was saying, you can easily imagine something like AB testing being applied to an e-learning course, for example, much more easily than a face-to-face training. It's much easier to change the color of a button. And then I suppose you also have to consider what the most success would look like in that context for a piece of e-learning. Cause it's something more fun and engaging, we want that to certain extent but not it's the point where it detracts from the content itself. So people are, I don't know, prefer a green button to red button. Does that actually tell you much about how they're engaging with that content
Also is that an experiment that's worth running? Because there's time and cost involved in experiments. And if their impact is actually not going to be that big a deal then, yeah. What's the point of doing it
Yeah. I think it's either it, to what extent are the results of the experiment going to inform your future practice in some way, shape or form? Whether it's going to determine which version you're going to roll out to a wider population or in the case that Ross has mentioned, it gives us a steer as to the things that do or do not make a difference. But what's interesting to me, there is, whilst you do need to do a little bit of extra work, I mean the impression I'm getting from you, Ross, is that it wasn't that much more work
It was hardly any work at all. No, we changed the subject line for 50% of users and then tracked how many completed it over the course of a month
Yeah. So I think all you need to do is to have a little bit of willingness to do that. And so we've done that, we will use the results of that to inform our practice both internally, but also it's a useful piece of information for clients to understand as well
It's also, in that case, it's been done by transport for London as well, and they got a different result. They found that having an authority source tell people to do something did make a difference. So it's not like, you want to take the context into account. It's not like the thing that I just said is applicable to every context
That's exactly what I was going to say, Ross. It's also not static. So you might get a different result in five years time or with a different CEO, for example, or someone else in the organization
Oh, that's a claim to make publicly
So we'll move on now to a regular feature of what I learned this week where we share something we've picked up in the past seven days. Owen, do you want to get us started
Sure. Well, this is kind of tied to the topic of the podcast. We have- one of our product teams adopted a new approach to how we organize the work. And it's worked, or it has eventually worked successfully in another team. But for this team, has been kind of like their first experience of using this particular method. And despite the fact that kind of upfront, one of the things that I said was this isn't going to go perfectly smoothly and we're going to have to iron out any wrinkles in the process, and don't expect everything to go perfectly
I think by the end of the sort of first cycle of this method, there's a sense of, Oof, right. We didn't deliver everything that we thought we were going to deliver. And I guess one of the things that I'm reflecting on is, no matter how much you kind of prepare people and kind of intimate it's okay if we treat this as a learning experience, if you don't deliver on all the things that you thought you were going to deliver at the outset, it can still feel like a bit of a failure. And actually, how we can get more comfortable with trying something new and it not going perfectly the first time. But treat it as a learning experience and go back and say, right, okay, what can we do to improve this process? It's a hard thing to do
Mm-hmm (affirmative). I think the thing that Thomke keeps landing on was the need to have the organizational culture to be able to do that. And that that doesn't happen overnight, that there's people do find it difficult, and they want to find a successful experiment. But if they only share successes, then there's a tendency of you not actually experimenting, you're just doing stuff that you know will work already
Ross, what did you learn this week
I have what the kids call a deep cut for long term fans of the podcast. So in episode 30, yes, January 31st, 2017, we launched an episode in which Owen, for his what I learned this week, recommended Simon Winchester's book, ""The Surgeon of Crowthorne: A Tale of Murder and Badness and the Oxford English Dictionary."" Do you know where this is going, Owen
No, but I thought the book was excellent
Yes. I remember you saying that because excellent. And so, it was about, for people who don't remember quite that far in the past, how professor James Murray collaborated with convicted murderer, William Miner to create a dictionary, the Oxford English dictionary whilst the murderer was incarcerated in a lunatic asylum. At the time, Owen expressed some concern that this book was about to be turned into a film starring Mel Gibson, oddly enough, in the role of the professor, not the lunatic
Well listeners, that film is now on Amazon video. Have you seen it
No, I've not
It is okay. Gibson is actually really good. Cause he needs to be like an outsider figure. He's like an academic, but not one with any qualifications. So he needs, I think they've deliberately gone for the counter casting of someone you wouldn't expect to play an Oxford professor, but who then comes in and has a success. Now I believe that since the film was released, both the Gibson and the director have disavowed it, and there were some legal ruckus over the final cut that emerged. There is some difficulty with some of the ethics speeches about God and truth and righteousness in the context of writing a dictionary. Which seems like they're going slightly too far. But I watched it last night, I thought Gibson was good. Sean Penn plays the lunatic in the asylum and yeah, it was okay
Yeah. Oh, well I just, couldn't imagine that converting well into film if I'm honest. So, I mean, even though it's on Amazon Prime and I've got that, I am probably not going to watch it
I would say that your concerns were well-founded, but that turns out Gibson shouldn't have been one of them
Great. Okay. Nicole, what did you learn this week
So my, what I learned this week is actually COVID-19 related. And since we are all now self isolating and social distancing, I heard on radio four today that someone very famous who was quarantined, wrote some masterpieces while he was quarantined. And that was William Shakespeare. So apparently, he wrote King Lear and Macbeth while he was quarantined and during one of the plagues. So yeah, I think we're all expecting great things from you, Ross, during your self-isolation and yeah, we can use it to our advantage and come up with something brilliant
No pressure. Yeah. Shakespeare did this. What have you done while you've been social distancing? Right.
Well, my what I learned this week is oddly related. I mean, I have not been writing..
He's about to produce a play.
I one-man play will be broadcast live on Zoom. No, I have not been writing a play, but as part of social distancing and self-isolation, I'm trying to think of ways to sort of look after our mental and physical wellbeing. And so my fiancee and I have taken up yoga. We've been doing yoga YouTube tutorials and I've never done yoga before in my life. And it turns out it is really hard. It's actually exhausting. It looks- people who are good at it are the people who are on these YouTube videos always look so serene. And as if they're not trying at all, but is actually absolutely knackering so. But yeah, I'm enjoying it and it's been a sort of good way to try and stay fit and get some exercise while we've been stuck at home
I would recommend the FIIT app. And, in association with that, F, double I, T. My wife, Amy is doing yoga teacher training qualification just now or she was, she had to travel for it. So last weekend, she did it online, but the app lets you get 75 lessons free and then I think you have to pay for it. But I also have found myself shaking and not breathing trying to do it
Yeah. That sounds familiar.
I think it's worth saying as well, lots of local yoga studios are also moving their classes online as well. So it's just another way to support them when you can't actually go to the studio
And that's all for this week. If you let share thoughts on the show, you can find us on Twitter. I'm @RossDickieEW, Owen is
Are you sure?
I'd forgtten I'd changed it.
And you can also tweet @Emerald_works. To find out more about Emerald works or performance support toolkit, custom e-learning and LMS visit Emeraldworks.com. If you enjoyed this episode, please subscribe and leave us a review on Apple podcasts, or simply tell someone about the show. Thanks for listening. Bye for now