Test, Learn, Optimize
Your resource to smarter customer experiences
Author
Seemingly simple ideas underpinning AB Testing are confusing. Rather than getting into the weeds around the definitions of p-values and significance, perhaps AB Testing might be easier to understand if we reframe it as a simple ruling out procedure. Ruling…
As AB testing becomes more commonplace, companies are moving beyond thinking about how to best run experiments to how to best set up and run experimentation programs. Unless the required time, effort, and expertise is invested into designing and running…
How often is A/B testing reduced to the following question: ‘What sample size do I need to reach statistical significance for my A/B test?’ On the face of it, this question sounds reasonable. However, unless you know why you want…
I was talking with Kelly Wortham during her excellent A/B testing webinar series. During the conversation, one of the attendees asked if they just wanted to pick between A and B, did they really need to run standard significance tests…
Multi-armed bandits, Bayesian statistics, machine learning, AI, predictive targeting blah blah blah. So many technical terms, morphing into buzz words, that it gets confusing to understand what is going on when using these methods for digital optimization. Hopefully this post…
In this post, we are going to introduce an optimization approach from artificial intelligence: Reinforcement Learning (RL). Hopefully, we will convince you that it is both a powerful conceptual framework to organize how to think about digital optimization, as well…
The key idea behind Conductrics is that marketing optimization is really a reinforcement learning problem, a class of machine learning problem, rather than an AB testing problem. Framing optimization as a reinforcement learning problem allowed us to provide, from the very beginning, not just AB and…
In our last post, we introduced the idea of shrinkage. In this post we are going to extend that idea to improve our results when we segment our data by customer. Often what we really want is to discover what digital…
As some of you may have noticed, there are often little skirmishes that occasionally break out in digital testing and optimization. There are the AB test vs multi-armed bandits debate (both are good, depending on task), standard vs multivariate testing…
UPDATED: 5/20/2021 A version of this post was originally published over at Conversion XL For all of the talk about how awesome (and big, don’t forget big) Big data is, one of the favorite tools in the conversion optimization toolkit,…