LISTEN TO THE EPISODE:
Present Your Split and AB Testing Results to Win
Valerie Kroll is a growing powerhouse in the measure and presenting community and in this show, she brings insight from a very specific and mystical arena of digital marketing, one I have been waiting for years to tackle on this show.
What is it, you might ask? How to present split and AB testing and conversion rate optimization results!
Valerie is passionate about data and she leverages her enthusiasm to help stakeholders make smarter decisions and mitigate the risks that come with digital experimentation and optimization.
As the optimization director at Search Discovery, she gets to spend her days crafting evidence-based hypothesis libraries and utilizing advanced AI testing platforms.
Her journey into Digital Analytics began in 2013 and I had the pleasure of meeting her when she was new to it all. Actually, you will enjoy hearing her take on our first encounter when we were both new in our respective fields.
Since then, she has soared.
She was a finalist for the DAA Rising Star Award and is an active member of the Digital Analytics Community.
If that didn’t keep her busy enough, she is also the current President-Elect on the DAA Board of Directors, a frequent speaker in the Digital Analytics Conference Circuit, and she appears on many podcasts.
Valerie is easily one of my favorite success stories and I am proud to present her as the latest superstar in my Women in Analytics Spotlight.
In this episode, Valerie discusses how to pass the test on presenting test results. More specifically, she answers several of my burning questions and shares her approach to building stories, dialing it all down, and working to really move the needle for her clients.
In This Episode, You’ll Learn…
- How learning to present data effectively propelled her from nearly quitting her job in market research to testing and conversion rate optimization management.
- What makes CRO different for her
- How she uses analogies to make her data more relatable to the audience.
- How to incorporate her “diet hypothesis statement.”
- How Valerie uses “if, then, because” statements and interactions to reach the audience.
How to Keep Up with Valerie:
People, Blogs, and Resources Mentioned:
- Tim Wilson
- Valerie’s ConversionXL Live 2019 Presenting Test Results session recap.
- Join Valerie and many others at Conversion Jam in October 2019.
- A/B Testing by Google, free course on Udacity.
Thanks for Listening!
Thanks so much for joining me. Have some feedback you’d like to share or a question for Valerie? Leave a note in the comments below, and we’ll get back to you!
If you enjoyed this episode, please share it using the social media buttons you see at the left of the post.
If you liked what you heard, I would love if you could leave me a rating or review in iTunes. Ratings & reviews are extremely appreciated and very important in the rankings algorithm. The more ratings, the better chance of fellow practitioners getting to hear this helpful information!
And finally, don’t forget to subscribe to the show on iTunes to get automatic updates and never miss a show.
A very, very special thank you to Valerie for joining me this week. And as always, viz responsibly, my friends.
Do you have a burning question for Valerie about presenting testing and conversion rate optimization results?
Lea Pica: [00:00:00] Hola, Lea Pica here. Today's guest is a woman in analytics who will teach us how to pass the test presenting test results. Stay tuned to find out who's making waves on the Present Beyond Measure Show, episode 45.
Lea Pica: [00:00:41] Hello-Hello and welcome to the forty-fifth episode of the Present Beyond Measure Show, the only podcast at the intersection of presentation, data visualization, and analytics. This is the place to be if you are ready to make maximum impact and create credibility through thoughtfully presented insights and data stories. And today you are here because you want to hear about how to present insights from a very specific and mystical field of digital marketing one that we've never talked about on the show before. And that is drumroll testing and conversion rate optimization. Woohoo.
Lea Pica: [00:01:21] All right so I am super excited about today's guest as usual. She is a growing powerhouse in the measure and presenting community and she's helping me tackle a topic that I have been waiting for years to be able to get out of in this show. So let's go.
Lea Pica: [00:03:03] She's passionate about data like really passionate and leverages that enthusiasm to help stakeholders make smarter decisions and mitigate the risk that comes with digital experimentation and optimization as the optimization director at search discovery. She spends her days crafting evidence-based hypothesis libraries and utilizing advanced A.I. features and testing platforms. So she's a little bit smart. She was a finalist in the D.A.A. Rising Star award. She's an active member of the digital analytics community and she currently serves as president-elect of the D.A.A. Board of Directors. She's a frequent speaker on a Digital Analytics Conference Circuit and appears on many podcasts just like this one. She is also the latest superstar in my Women in analytics spotlight and happens to be one of my favorite success stories. So please help me welcome Valerie Kroll.
Valerie Kroll: [00:04:06] Hello. Hello. Thanks for having me. I'm really excited about this.
Lea Pica: [00:04:10] Me too. We have had so many interesting intersections in the last few years. And what's funny is we didn't actually meet the first time we made a connection. So I would love if you actually told that story because it was more from your perspective.
Valerie Kroll: [00:04:29] Yeah absolutely. So I got into digital analytics in 2013 and in my first role I had some difficulties. So it was back in 2013 and I was having some difficulties in my role just because of where my new function was positioned within the organization within I.T. and some people who were my stakeholders had applied for the role that I had been given and so I wasn't necessarily chosen for success. So it was a little bit of a difficult start and I transition for market research and so there was a lot of ramp up not only on how to approach this whole new way of leveraging data for insights but the technology piece and just the challenges that come with trying to ramp up in a new industry. So. I went to accelerate the conference in 2013 and I think that that was one of your first presentation.
Lea Pica: [00:05:29] It was my second one ever.
Valerie Kroll: [00:05:33] You were still at Prudential at the time.
Lea Pica: [00:05:34] I was. Yes.
Valerie Kroll: [00:05:36] And you gave a presentation about your journey with presentations and how it had evolved over that time and it couldn't come at a better moment because it was not you like I said I had some difficulties within my role and the stakeholder mind was huge and so just that new lens of how to approach presentations was really meaningful and I think I told you this before that in that presentation you had little light bulbs for every time there was a resource that you were recommending a book and I purchased all five of those books before you started with the Q&A.
Lea Pica: [00:06:14] You're welcome to Amazon.
Valerie Kroll: [00:06:16] Exactly. So I felt like I tried to replicate as much of your journey because I'm not advising it was it was really impactful and I think that attending that accelerate conference was really meaningful to my career because that was when I first had my in-person experience with the digital analytics community. I really hadn't been exposed to it too much before then and I was like oh my gosh like I'm not alone. Like other people are struggling with some of these things too. So I really felt like I met my tribe and my people. So it was that and then metrics three weeks later and I was like You know what. I'm sticking with this like you were going to do this.
Lea Pica: [00:06:55] That is so amazing. So thank you to analytics demystified for hosting accelerate that year. And thank you, Tim Wilson, for dragging me to speaking but I mean I think you had reached out some time after. Or perhaps we encountered each other at a later event and when you told me your story you know it meant so much to me at that time because the second time I presented I was not convinced that people wanted to hear what I had to say or that they didn't already know everything that I had to say and I was just blowing smoke and hearing your story really made me feel like all those years of kind of struggling and tearing my hair out over trying to figure out why I could not like to make affect real change for myself and my career and then when I heard that I was like Oh this is it. So I've just been so grateful for how we've intersected the last few years and it's been amazing to watch you grow and flourish especially in an area where I think people are clamoring. For information. But before we get to that core topic first I'd like to understand you did present feel like it came naturally to you. Was it an evolution and how has that approach to presentation evolved as you've really honed your craft.
Valerie Kroll: [00:08:21] Yes absolutely. Even though PowerPoint had been my primary delivery mechanism for communicating to clients back when I was in market research before the script that I used to put together those presentations didn't necessarily translate to the new work and digital analytics and I didn't expect that I kind of saw a lot of similarities between my role in market research and what I was doing in digital analytics but it was a totally different narrative. So in market research you always talk a little bit about your methodology first like that's not an appendix while you're like you know this was a phone survey or this was in-person ethnography is or a focus group and that really helps that context. And then you go through the story linearly like you talk first about unaided brand awareness and then aided brand awareness and and telling it that way is how you communicate a message whereas in digital analytics like people are more focused less on like OK so we place these tags and then we collect the data like no one wants to hear about just how complex it was your idea and not to say that that is how I started my presentation but don't worry it wasn't that bad. But I thought that that context is like starting at that place was the story that my audience came to hear.
Valerie Kroll: [00:09:44] And I quickly realized that that was not what was going to resonate that I needed to really switch to focusing more on the narrative that was going to resonate with the market or with the product team that needed to be translated. And I didn't need to educate people on how to collect the data or how to become an analyst or a CIO that it was more about how do we couch this in something that's really meaningful for you in the presentation pardon and the public speaking part or you know holding the floor so that you can tell people about how data interject into your strategy is something that I think not naturally because I definitely think I've grown into this over time but the part that I feel the most change is the way that I approach building that story and the way I edited it down to just the parts that are really going to move the needle.
Lea Pica: [00:10:36] Well that's interesting. So how big of a role would you say you know presenting now the insights? How big is that compared to past positions that you held?
Valerie Kroll: [00:10:49] Yeah I think it's still a major part of what I do. But the positioning is totally different. So in my new role a search discovery I'm presenting to my clients and oftentimes I'm trying to give them the information that they can pass along to their internal stakeholder's point. So sometimes we give them a little bit more of the things information because we want to make sure that they're armed with that backstory but making sure that we're supporting that scenario box lungs a lot of the times were in my past roles as a practitioner.
Valerie Kroll: [00:11:20] I was giving that end-user presentation. The majority of the time and so being in house and being really tightly wound into all the moving parts and pieces of the strategy and you know that new email that came out from the CEO of this morning and so I could always pepper those things in the presentation. So I think that it's one layer removed. And so it's kind of it serves a different role in purpose but it's still the majority of where we're spending our time.
Lea Pica: [00:11:47] So are you finding it challenging if people are if things are getting lost in translation when they're taking your information and disseminating that in their own organs and organizations.
Valerie Kroll: [00:11:59] So that isn't something that I've run into you with my current clients so much because again there was like prep sessions or those like pre-presentation sessions more more more showing them how are crafting that narrative. We're very careful to say or to even have a conversation about what belongs in that primary narrative that primary docs and what we can hold back from that story. So I think that they do a good job passing that along to their end users.
Lea Pica: [00:12:28] So it sounds like the great work that you're doing is not just handing over a story that makes sense to you as a practitioner but you're really fleshing out from their own needs perspective and what it is they want and what ultimately really matters. You know you're separating the wheat from the chaff before they have the chance to go and run with it. And I think that's an amazing preemptive action you're taking on here partly because it could really go both ways. So I want to dive into the core topic that you are like really rocking right now. So I heard you absolutely killed it at the Conversion Excel Conference earlier this year. And I'll even admit I'm just really now this year preparing content around presenting optimization results because I don't actually have a deep background in testing except for email marketing which I had played around with. And I think I've come up with some great stuff but I thought who better than to bring on someone who applies all of the best practices of Data storytelling and visuals but is also focusing so much on this very important niche essentially of digital marketing in itself. So first I want to start with what do you think makes presenting CRO data different than other kinds of general usability or Web site or email marketing or search marketing.
Valerie Kroll: [00:14:01] I would say that the biggest departure is the sophistication of the analytics in the statistics at the time so it's really easy for someone to watch to your example like usability research like an unmoderated study.
Valerie Kroll: [00:14:16] You can watch that and be like OK that's familiar. I understand what that is. Or if you're looking at a trend on purchases or conversion rates from website metrics that's really easy to understand what that's all about. But when someone's looking at slides and they see a 1.3 percent conversion rate on your control and 1.5 percent calendar and sometimes that's a significant meaningful difference and sometimes it's not all of a sudden the conversation is like well what's a p-value and what are common goals. And so it gets it can go deep real quick. And so I think that's what makes some of it a little bit more complicated or tricky at times.
Lea Pica: [00:14:59] So I'm glad that you brought that up p-value in terms because this is also a lot of the questions I get from my students is how do you make concepts like statistical significance the confidence level things like that more accessible to your lay audience. Because for me what I found is it's important to express that we feel confident enough to take an action without having to give them an entire college freshmen session of stats. But I'd love to hear how you keep your audience out of the weeds on stuff like that but they still find that they can trust what you're what your direction is for them.
Valerie Kroll: [00:15:41] Yeah and I totally agree with you that giving someone an education session especially during a briefing is like the biggest no-no.
Valerie Kroll: [00:15:50] I actually learned that trial by fire. I was the first time testing was a part of a major initiative when I was at the American Medical Association back to the beginning of my Digital Analytics career. I was going through some test results and there was a huge digital strategy team and audience and there were some questions about some of these results and I took that as a cue to educate on these concepts and I was up at the whiteboard in no time like drawing normal curves and shading things in you know boy was over like I totally lost my audience and missed my chance to talk about the test.
Valerie Kroll: [00:16:26] That was a winner that we did have a significant change for and on the way out. The CEO said he was like you're telling me all about how to make the watch. But I just want to know what time it is. I was like whoa burn me, okay.
Lea Pica: [00:16:40] That's an amazing analogy for that.
Valerie Kroll: [00:16:44] Yeah that was big for me. And that's when I realized like how much more I needed to be changing a story as it relates to presenting test results and so one thing is to first remove those educational moments and those educational sessions from the presentations themselves so you can have those conversations about what a significant changes and how you'll know when to make what choices but don't steal your time from your results presentation like try to completely separate those two concepts.
Lea Pica: [00:17:16] OK.
Valerie Kroll: [00:17:17] So that's one and then second in the presentation I don't think you need to talk about any of those statistical concepts because the reason everyone came to that meeting is because they're so excited for new information that's going to help them do their jobs better or make a smarter decision it might lead to more impact. So I think you can leave that out completely. If some of that needs to be there because you know you're going to get that question. That's when I would recommend leveraging the appendix to see you can bookmark your narrative where you're at. You can flip to that to dig into whatever questions but come back to the story that you prepared. And when you're forced to have those conversations in the presentation I recommend using analogies as much as possible versus like pulling up an equation and making someone feel like you're back in high school or something. As an example, if you're trying to explain the concept of confidence that's one that's really commonly misunderstood because people think that that's the confidence that the results are real but the best analogy I've heard on that when I'm not taking credit for this one at all. So when someone passed this along to me.
Lea Pica: [00:18:23] That's OK it's all being passed along and.
Valerie Kroll: [00:18:27] Right. All sharing. So if you went back in time and a time machine and re-ran that same experiment ninety-nine more times how often would you expect the same outcome and so that that helps people tease apart the different the confidence and the result and what confidence represents. And so it does get like you know kind of it can go off the deep end quickly and so again that's my defense or my recommendation. And again if the goal of the presentation and your time together with your stakeholder audience is to describe what we learned and to help people outline the recommended next steps and action then. Ideally, you can stay out of you know what that 90 percent or 95 percent significance.
Lea Pica: [00:19:14] That's fantastic and I love that you're like one step ahead of me because my next question was literally do you ever use analogies because I'm actually planning on gathering lots of analogies from you know thought leaders because I feel I'm 100 percent with you as soon as you've really a technical concept to something in the real world that people can relate to. It's like this. You can see a light bulb go off and I remember that that is how everything clicked for me in digital marketing. I was fresh at an agency. I felt like I was drowning in like technical stuff and I just wasn't getting it. It was so frustrating. And then someone sat me down and described how marketing tracking pixels are like pieces of paper being passed back and forth. And like my job, because it was like It's so simple why did I not understand.
Lea Pica: [00:20:11] But it is such a powerful tool so do you have any other analogies to share that you ever use like that because that was I love both of those.
Valerie Kroll: [00:20:21] Yeah that another one that is really common is the difference between type 1 and type 2 error. So understanding the relationship between those two and how when you're planning for reducing the error on both sides how those are inverse relationships. One of my favorite ones there is putting the guilty letting the guilty man go free or that innocent man in jail. So it's understanding like were you able to see a difference when there wasn't really one. Or did you not call it when or when there actually was. And so that's really what it boils down to. But when you get into like Alpha and Beta and you know all those dots that's where it loses contact. So I think that that's the best method.
Lea Pica: [00:21:03] See already my brain almost went abort with all the technical stuff. But as soon as you mentioned that story which I'm familiar with I was like Oh I feel like I could totally get that. So powerful I love it. So one of the things that the Conversion Excel people loved about your talk was you have this template formula for not only presenting the business case for why someone should buy into a test because that's something interesting here too in digital marketing. A lot of times people were like oh you're just setting up a search campaign or we're building a website, just build the metrics and your present kind of after all of that's happened. But with testing, you actually have to get buy-in to perform the test so you have to sell them on the idea of the test and then sell them on the idea of the results I see that is two discrete events right. Absolutely. So I would love for you to talk a little bit about the formula you have for actually pushing getting buy-in for test cases.
Valerie Kroll: [00:22:11] Yeah. So I'm going to go back to when I think about what this formula is or when I sort of put this together. It was actually a piece of advice that I heard from you at that first conference. Think it really hit home when you said your slides or not your presentation you or your presentation. That's right. The slides are nothing more than the visual assist to the message that you're trying to deliver home plate and butcher that.
Lea Pica: [00:22:40] You're almost there and I just love that it's getting passed on and I have to add it back to my keynotes. Now I just realized.
Valerie Kroll: [00:22:48] But if I'm using that as the lens then I'm focused on like wanting to inspire action and that conversation that my stakeholders are going to have with their leaving the room. It's really about why we tested and studying your business case first. And so like how did this idea come to be like Why is this an opportunity we should take advantage of and talk about any evidence that you've gathered either quantitatively or qualitatively that support that this is the right next move. And I think even when you're in your results presentation mode starting with that context is really good because sometimes these tests run for six weeks a field and think about how many things change you know especially for someone who's not in who's a stakeholder test who's not in it every single day monitoring to make sure things are going well.
Valerie Kroll: [00:23:36] They might have lost some of that context. So I'd love to restart and refresh with that and then digging into like what was tested just like right out of the gate. And I love putting visuals here. I love titling that slide with what's called like I've dubbed the diet hypothesis statement diet hypothesis.
Lea Pica: [00:23:56] Ok. What's that?
Valerie Kroll: [00:23:57] So it's like the really succinct question that you're going to solve with this experiment instead of like you're full-blown.
Lea Pica: [00:24:05] Loaded.
Valerie Kroll: [00:24:06] Exactly. Yes. So it makes it a little bit accessible but by starting here with that simple question that's like really encapsulating the what you're trying to drive forward the risk you're trying to mitigate the new feature you're trying to get a read on including the visuals that really route it and something familiar because sometimes the concepts are what you're changing again not everyone in the room is going to be in all those details. And so if you can give them something familiar then they can relate early on and they can get onboard easier.
Lea Pica: [00:24:40] And how do you present the visuals you said you love to start with visuals up front to give context for the test. What kinds of visuals do use?
Valerie Kroll: [00:24:48] Screenshots of what the experience looks like and this isn't like the time to take screenshots across all different devices and put them like well.
Valerie Kroll: [00:24:56] Throw that in the appendix too but make it really clear what was changed was it copy? Was it a flow? Like think about if you need to use arrows or call outs to really show and visualize what was altered then maybe you're sending people down in different path because you're going to auto log them in so you can show the little jump with an arrow over in the flow but try to make it as big as possible make it easy for people to see what was different between the variance is so great.
Lea Pica: [00:25:28] Now in terms of presenting you talked about why they should do the test. So I'd love to know I've been kind of testing two different ways of going this way a benefits-oriented argument. What you'll gain versus the actual opportunity cost if you don't take any action. So I was wondering do you ever run projections that show either side of that coin.
Valerie Kroll: [00:25:55] I think that that one so two answers there for you. One is I try to stay away from that potential incremental impact or lift in terms of a dollar amount or something especially as you're presenting those results because those are estimations anyways and how you get to those estimations is like a two hundred level concept sometimes.
Lea Pica: [00:26:18] Ok.
Valerie Kroll: [00:26:19] If you can avoid it sometimes you can't. I try to keep out. I think that that's appropriate for your prioritization criteria. This is Library but not something you want to remind someone of especially if the outcomes or the way the test was designed was slightly different than it was originally conceived. The other thing is the benefit statement versus like de-risking like if you're like I just want to make sure that this alternate variation doesn't hurt conversion because that is the outcome of that test. And so some of that differs whether or not this is more of a marketing type of test or if this is product. So the business of injuries as validation or is this more a new opportunity. Messaging and promotion flow that type of thing. Or even UX that even might be a third bucket if you're altering the frictions in the path that you're taking people through. So you could alter it all different ways just depending on the type of experiment.
Lea Pica: [00:27:21] Oh cool. OK. So in terms of stating the hypothesis, two audiences tend to get lost at all when you're using kind of fancier and I guess it might depend on the level of savvy of the client. But is there. What are some of the ways that you state hypotheses and you know what the KP eyes are? That helps keep people on track for understanding like the eye on the prize.
Valerie Kroll: [00:27:48] And that's actually the next slide. The next piece of the template is to line out or to detail out rather the full-blown hypothesis statement. And underneath it to show what your primary success metric is and then all the other things that you're going to drill into or dig into as a part of your evaluation criteria of whether or not this is going to be a successful outcome however you've defined it. The most accessible template or format for hypothesis statement that I've found is that if then because. OK so if and then insert or describe your variance then and then that's where you talk about your expected outcome or the impact on the KPI. And then because it's like your rationale statement like why is this going to win or what. What are some of those evidence pieces? And so it's a nice sentence structure and you can even use that in your hypothesis library for columns of if then because you can detail that out early on your test planning thesis you can really think about what you want to alter and the impact you expect. And so the metrics that you derive from that are really easy because it's a solid line from what you put in and then part of the statement if it's increase in leaves captured or increase in the number of purchases then that becomes your primary KPI using your t-test the type of experiment.
Lea Pica: [00:29:11] So can you give us an example of an if-then-because statement that you might have created recently and of course don't share client details or anything like that but I think it would be great for us to hear what a real-world example might sound like.
Valerie Kroll: [00:29:26] Sure. So a recent example was testing copy of a CTA button for it was a video in front of so some content for one of our healthcare clients and so it was if we changed the copy to XYZ and talked to us I think then more people will click through to our video form becomes this copy is more conversational.
Lea Pica: [00:29:54] Got it.
Valerie Kroll: [00:29:55] And they also we found a past winning Test where that similar messaging strategy worked in email and so we took that learning from the other channel and tried to take that to other places to see if that's where we could get some changes in user behavior.
Lea Pica: [00:30:09] Oh that I see that crystallized it for me. Exactly. That's really cool. And another strategy that you offer and I love this one I definitely need to do this more of this in my own talks but is for making your presentations more interactive so what's one of the ways that you're able to do that.
Valerie Kroll: [00:30:27] Yeah. So after that's actually a very nice part of the template.
Lea Pica: [00:30:31] I'm not following it at all.
Valerie Kroll: [00:30:36] So after you've laid out the business case you given them the primary question you got into the details of how you're going to measure success in the details of the measurement plan. I like to make it interactive by asking people either live in the meeting or before the meeting which variation do you think one. It does not replace a well-thought-out evidence driven hypothesis statement. We're not just taking shots in the dark. This isn't what do you think. But this is a good moment to just pause and have people give you their answers and their rational statements and why they think it when and I like this for a couple of reasons. One is that if people in the room are going to have to make a call. we're asking them to make a call. They're going to have to think are there any other pieces of information I would have to have to answer this question. And you can clear that up before you even get into the results. So before people are looking at the outcomes they can ask well did you run this on mobile too or how did this look for people who are already customers.
Valerie Kroll: [00:31:44] So there are any of those details that you could flesh out or describe before you're looking at results. Then it doesn't become about the validity of results when they're asking me questions. It's just interrogation so that they can be really following along in the story. And the other reason I like it is I started doing this just for fun the first time we ever ran a test at AMC and we continued with it and never to this date. Knock on wood. The majority voted for the winning variation.
Lea Pica: [00:32:15] Isn't that amazing.
Valerie Kroll: [00:32:17] And it's such a good marketing point right. Because it's like if we had all gone with our collective gut. Yep we would have done some negative business. That's not to say that there are some people who are good at guessing the winning outcome and you can get off by that and make it fun.
Valerie Kroll: [00:32:31] But it's just a really good proof point. I always collected that as part of the information of our program help score is the number of times the majority voted and we always saw the zero there.
Lea Pica: [00:32:42] That's awesome.
Valerie Kroll: [00:32:42] Another touch point to be evangelizing why we test.
Lea Pica: [00:32:46] I love that actually as a metric for the tests that you as part of your scorecard you actually have accuracy of Team weighing in. And this is again why as many times as people asked me like what would you do and what does your gut say I'm like and I don't trust that at all because there used to be a service. It might still be around called which test one it used to be that they like my favorite time of day when I would get their email and basically they would email you the results of a very potent testing case study and they would ask you to predict the winner and I am telling you my accuracy rate was like 3 percent. Every time I'm like This is it this. No. Every single time and it's just amazing because we are simply not our customers. It's rare when we're actually measuring something where we are the exact customers and sometimes even having that meta-understanding of marketing completely shifts our perspective on how real world people are going to behave. So I love those tips about because it adds a layer of gamification in there too right now.
Lea Pica: [00:34:03] Do you have people predict a winner when you're both selling the test idea and also presenting results?
Valerie Kroll: [00:34:12] So it's usually not when I'm filling in the idea although my brain is thinking through how you could potentially involve that in my past roles as a practitioner. I have been above no tactic to get people excited about strengthening to this. If that could work I would be all about it. It's more something that we've communicated as soon as the test launches so there would be an email communication or there'd be some forum for the testing team to say like hey guess what. This is life. You might see this one when you're hitting the website. But also get excited. We might be able to learn something new here. And so I would use Outlook voting button but sometimes communication or sometimes we do like whiteboard tallies just different ways in any way that you can get people kind of excited or interacting with that. So it could come in at any point but if you live in the meeting you know you could take like a show of hands and especially if there's a hippo in the room especially if they're wrong. It's a great little fun light moment.
Lea Pica: [00:35:18] Everyone makes light of it. Yeah. Oh, that's fantastic I really like that. All right. So finally one of the key parts of your template also seems to be how you present learnings in action the so what part. So I would love for you to go into that a bit.
Valerie Kroll: [00:35:35] After you talk about your primary KPI and you dig into your segmentation and you walk everyone through your analysis. This is the slide but like the hit at home so if I'm thinking about the agenda of the time I want to spend together of my stakeholder audience I want 40 to 50 percent of my time to be. With this slide in the background because this is where it all the rubber meets the road. So I like to present this side by side visually so that you can see how every learning ties to an action step because a common misconception is that you're only learning are moving things forward. If you have a winning test that is absolutely not the case you can learn something from every test outcome whether it's a win or a loss. Or if you want to communicate that save. So some other learnings might be OK. This one didn't win and the actual step is so we're going to leave the current control in place but then the next step might be. But we don't feel like this idea is done like maybe we didn't put our best foot forward so we learned that this the way we put this Test forward didn't make a change in user experience.
Valerie Kroll: [00:36:44] So we're going to go back and talk with the marketers or we're going to do some usability research. We're going to drill into some additional reporting to see if there's another opportunity. So again it's all about what we learned and how we're going to tie that to our next steps once we leave this presentation.
Lea Pica: [00:37:00] Okay. And when you say you present them side by side is it like paint a picture for me visually what that looks like.
Valerie Kroll: [00:37:09] So I have 50 percent on the left-hand side tied to a list of learnings and then they line up to the other half of the slide which is the action statement. So there I know you could put it in a table.
Valerie Kroll: [00:37:22] Sometimes there's more than one action coming out of a learning meeting maybe new test ideas that you want to go back into the hypothesis library for prioritization because a lot of new test ideas come from things you've tested in the past so it could be a table. But I love looking at it side by side.
Lea Pica: [00:37:40] Awesome. Okay, so we are at the question that I've been dying to ask this whole time which is how are you visualizing the actual differences in your AB or ABT testing results. What kind of charts are you using? How are they different than a normal chart you might see in a regular SEO presentation, I'm dying to know.
Valerie Kroll: [00:38:05] Well it looks very Lea Pica-esque all right though it's the out of all the slide in the template, this is actually the one that I suggest being the most prescriptive on if you're going to try to use this for the way that you present test results so the headline is going to talk about your single sentence impact on the primary KPI and that's going to be directly tied to a single visual which might just be a bar chart and a lot of cases maybe it's just too hard to keep it simple.
Valerie Kroll: [00:38:40] And then beneath the actual visualization is that little bumper that you have on some of your slide templates and that's what I call the translation bar.
Valerie Kroll: [00:38:49] So that's where you take the maybe it's a twenty-five percent lift or you know your challenger had a 2.3 percent conversion rate and that's where you take those numbers that are really meaningful to analysts and the zeros and you translate to something that's meaningful to your marketers or your product team so maybe you annualize that impact and you say this means this year you will get you to know five thousand more leads or if we were to project this out across all the different times we run this campaign you would receive this much more impact on X Y Z. So it's where you take the numbers and translate it into something that really resonates with your what they're saying. The reason that you ran the experiment.
Lea Pica: [00:39:35] I love that so much. It's kind of almost the opposite of taking a projection of what you either might gain or might leave on the table. But I just actually did a testing oriented Pica Protocol keynote for Brooks-Bell, great testing firm and I found that one of the more powerful things I just threw in there at the last second is what this actually translated to a translated to like tens of thousands of more applications for something and making that more concrete taking that out of just a percent lift of something that seemed to be the thing where people were like oh OK so again bringing it right back to that real world concept I think is so important.
Lea Pica: [00:40:22] So that's really, that's really cool.
Valerie Kroll: [00:40:24] How did you visualize that for that team?
Lea Pica: [00:40:27] So I actually I went through a couple of different routes in terms of visualizing the difference in performance between the two test cells. I used a bar to her that's why I was kind of relieved because I'm like Is there really a fancier chart here that I'm missing. No. This is what works. So I did kind of a combination and it's a work in progress but first I visualize two of. There KPI is as bar charts and just show the comparison. And then I made sure to put a pretty prominent kind of label near the winner saying the confidence level so there was like some kind of visual cue that not only did it outperform but this was a significant winner. And then what I did is I created a variance bar chart were for those two KPI's. I put the two I put the winning cell's performance in one graph and I set the zero access to the control performance for each of those caps eyes and then for each of those each bar represented a separate measure like click through and then application complete right. But each of those performances was how much higher the winning version was over the control. So it's not necessarily perfect yet I'm still trying to find two and making sure that that's really understandable. But it seems like a creative way to represent not only because again like you said the performance looked like they were pretty similar. So seeing it that way might surprise people that it was a significant win but seeing it as variance seemed to be more like oh OK I see because this is you know this much higher. So.
Valerie Kroll: [00:42:22] I like that.
Lea Pica: [00:42:23] what do you think? You're the expert.
Valerie Kroll: [00:42:26] Yeah I like to do the single slide on the primary KPI because that's the signal that you're going to use for your test. And then after that is where I typically suggest the drill into any of the secondary Keep your eyes or any segmentation you looked at or the down funnel downstream metrics. But what we talking about here is like one visual for that whole cannibalization analysis right. So this challenger outperformed the control on our primary KPI but let's make sure it wasn't Dr. Fife got everything else we consider valuable. So I think that that can be a really cool way to to do it to drill into that.
Lea Pica: [00:43:04] That's really neat. And what was interesting I tried to leverage some storytelling techniques like surprising twist turn turns of events. And so I incorporated things like Well what surprised us about this. And that can create some anticipation and you'll you can further visualize like well this metric actually underperformed in this metric. We don't exactly know why yet but it's definitely something we're going to look into and we can account for here and here. So that way you're interrupting kind of that linear flow of Oh this one won and then the book is closed on the story.
Valerie Kroll: [00:43:41] Yep.
Lea Pica: [00:43:41] Neat. All right.
Lea Pica: [00:43:45] So I call the next segment the upgrade which is a power tip for doing our jobs of presenting data more effectively it's a resource it's a tip it's a book something to do your job more awesomely so what you have for us.
Valerie Kroll: [00:44:02] Yeah. So a lot of clients are asking about how they can skill up on the statistic side of zero because they're familiar with the way that the data is collected and how they're testing tools are splitting traffic but digging into some deeper concepts especially as the tools get more and more sophisticated how can they ramp up on that. And the best resource that I've found is there's a less free course on Udacity that they did in partnership with Google with a Google statistician and a Google. And it's like a 14-hour course it's like the little parts and pieces that don't get you into me because you can pick and choose what you want. You can drill in on any different specific parts and there are exercises and they walk you through it just like in those Udacity courses. It's all free. So. I recommend that.
Lea Pica: [00:44:52] That's incredible. That sounds like a course I needed a long time ago so I'll definitely be checking that out and we'll make sure the link for that and everything we've mentioned is on the show notes page for this episode. I also wanted to throw in a couple of questions before we get really wrap things up. What's your dream test as if you could test anything for anyone. Sky's the limit. What would you do?
Valerie Kroll: [00:45:16] Oh that's an interesting question. It's to me it's less about what the test is and more about like finding a way to that test is so deeply ingrained with business impact. So before I joined search discovery my goal was like let's deploy the million dollars so that the outcome within that test period made a million dollars of impact. And so it's not going to be a button color that's going to get you that million dollar impact it's not going to copy tweaks you know deep below the fold it's going to be about business level testing and so I get really jazz and excited when we can say that we're going to run an experiment on something that would be transformational to your business because a different approach to how you're talking to your customer. So the way you interact with your customers which take more time usually for not only the buying and approvals but for the development but that's what really gets me jazz is when you can do those business type testing that's really going to change the direction.
Lea Pica: [00:46:21] So something transformational might be like an insurance company and you want to get applications in and won control is all you've ever done is had a Web site and there's an application that they can find somewhere versus having a chat but some sort of a high powered chap come in and actually enhance the experience for them as soon as possible and that sounds like a more dramatic way of changing the experience.
Valerie Kroll: [00:46:49] Yeah. And other applications of that is personalization. So that you're not just using pixels to talk to customers in a personalized way but what are ways that you could take that to brick and mortar or to the kiosk or to some other locations that are usually considered out of scope for some traditional testing experience but could be a big game changer for how you talk to your users.
Lea Pica: [00:47:17] That's awesome. Well, thanks for asking that one kind of popped in my head. Well, we were talking and really that is something really to think about we do often get caught up in tiny ammunition I think part of that is they're the easiest variables to control for. But you're right it reminds me of this author named remit city who teaches people how to become wealthy by their own definition. And he just had an e-mail where he said No I'm not going to tell you what brand of fabric softener to buy so you save 13 cents a month I'm going to help you make sweeping changes to and amass great amounts of wealth by making big changes to certain things so that that resonates. All right.
Lea Pica: [00:48:03] So this is our final question. Think very hard and imagine this very plausible scenario, I'm already laughing.
Lea Pica: [00:48:12] You are just leaving a 90s throwback hip hop yoga class with me because those are my two favorite things ever. I laughed when I saw that. When suddenly you trip and fall into a vortex that pulls you back to the moment you're about to deliver your first presentation. What are you presenting about and what would today you say to yesterday you.
Valerie Kroll: [00:48:38] So back to my first presentation I remember this actually pretty well. It was my first job out of college in market research. My client was Time Warner Cable and I was about to tell them that their customer service. Ah yeah. Their C SAT scores had dropped significantly. So it was a difficult message. And what I would tell myself is prior to going back is to relax and do enjoy it because I treated those presentations like us preparing for a court case like I had to submit all these different pieces into evidence and it was I was going to get interrogated by my audience. But the reality is that that's not always how it goes balance usually not how it goes down in that my analysis and the rigger that I put behind it was enough that you know it's okay to say you know actually I'm not sure on the following question. Let me take that back I'd be happy to research and get back to you. But that is a perfectly acceptable answer and you don't need to. You don't need to have all the answers in the meeting and kill yourself and not only in preparation but in terms of the nerves. And so I think I would give myself a little power back in hopes of upping the competence and just say enjoy this moment because you're going to end up loving presenting data and results to people for a long time to come.
Lea Pica: [00:50:06] I think that's a really valuable piece of advice that hasn't really been talked about yet on this show especially because I feel that sometimes we're expected to be like Alexa.
Valerie Kroll: [00:50:18] Exactly.
Lea Pica: [00:50:19] And we are not, we are human beings. We have. We haven't taken the limitless pill. And I think when we are prepared for the idea and even say Oh I hope I get a question I don't know the answer to because that gives me a chance to follow up with my audience offline and build a relationship with rapport with them which is going to help my career. You know completely reframing that from going I'm a failure because I didn't know to. This is an opportunity can really help you because the only person that can believe convince you that you're a failure ultimately is yourself. So if you don't feel that way you won't feel that way.
Lea Pica: [00:51:07] That's came out differently. Well, Valerie, this was really special for me. I've loved having you on especially because we were talking about a topic that's really been hot off the presses and so many folks want to know how to present this kind of unique interesting information in an effective way. So tell the listeners where they can keep up with you.
Valerie Kroll: [00:51:32] Yeah absolutely you can find me on Twitter. valkroll. All lower case, shout out to Tim Wilson there, he always teases me. And I'm also really excited to be presenting about this exact topic at Conversion Jam in Stockholm and Oslo in early October.
Lea Pica: [00:51:52] Oh that sounds great. Awesome. Well and all of the links once again are gonna be on the show notes page for this. So, Valerie, I hope our paths cross again soon. I love it when it does. And in wishing you all the best.
Valerie Kroll: [00:52:08] Thank you very much. Thanks again for having me. This is really fun.
Lea Pica: [00:52:15] Well, that was awesome.
Valerie Kroll: [00:52:18] You know it's really been amazing watching Valerie really rise up and make a name for herself in this industry. After we initially met and I just think that the knowledge that she shared today in presenting a very unique style of data that's within our field can be really challenging and I hope that you've walked away with something really practical that you can use starting tomorrow. So to catch all of the links and resources mentioned in this episode please visit the show notes page at leapica.com/045. I would love if you could leave me a comment or suggestions because I want to hear about the challenges you face when you're presenting information and trying to get your insights noticed and acted upon.
Lea Pica: [00:53:03] And today's presentation inspiration is by Joseph Chatfield. And that is oratory is the power to talk people out of their sober and natural opinions. And we've come a long way since oratory but my take is that presenting testing data is definitely one of the more powerfully convincing areas of digital marketing that you can use to help your organizations become more data-driven and less gut-driven. Now presenting testing data well is what I hope to help you continue to do on your path to becoming indispensable.
Lea Pica: [00:54:44] That's it for today. Wishing you an awesome June. Namaste and Now I'mma go.
