Test, Learn, Optimize
Your resource to smarter customer experiences
Content Type
As AB testing becomes more commonplace, companies are moving beyond thinking about how to best run experiments to how to best set up and run experimentation programs. Unless the required time, effort, and expertise is invested into designing and running…
I was talking with Kelly Wortham during her excellent A/B testing webinar series. During the conversation, one of the attendees asked if they just wanted to pick between A and B, did they really need to run standard significance tests…
Multi-armed bandits, Bayesian statistics, machine learning, AI, predictive targeting blah blah blah. So many technical terms, morphing into buzz words, that it gets confusing to understand what is going on when using these methods for digital optimization. Hopefully this post…
In this post, we are going to introduce an optimization approach from artificial intelligence: Reinforcement Learning (RL). Hopefully, we will convince you that it is both a powerful conceptual framework to organize how to think about digital optimization, as well…
The key idea behind Conductrics is that marketing optimization is really a reinforcement learning problem, a class of machine learning problem, rather than an AB testing problem. Framing optimization as a reinforcement learning problem allowed us to provide, from the very beginning, not just AB and…
In our last post, we introduced the idea of shrinkage. In this post we are going to extend that idea to improve our results when we segment our data by customer. Often what we really want is to discover what digital…
As some of you may have noticed, there are often little skirmishes that occasionally break out in digital testing and optimization. There are the AB test vs multi-armed bandits debate (both are good, depending on the task), standard vs multivariate…
Personalization, one to one, predictive targeting, whatever you call it. Serving the optimal digital experience for each customer is often touted as the pinnacle of digital marketing efficacy. But if predictive targeting is so great, why isn’t everyone doing it…
Normally, when we talk about AB Tests (standard or Bandit style), we tend to focus on things like the different test options, the reporting, the significance levels, etc. However, once we start implementing tests, especially at scale, it becomes clear…
While AB Testing is currently the default method for optimizing online conversions, there are frameworks other than AB testing for selecting between competing optimization options. One of the main alternative approaches is known as the Multi-Armed Bandits problem. Solutions to…