Having been a B2B PM all by career, I am an A/B testing dunce. I know a few concepts, but I’d really like to learn this properly (and quickly, in preparation for a new role)
Much Thanks.
Study about “randomized controlled trials”, since this what A/B tests are. There’s plenty of free resources online. Familiarize yourself with product execution case studies. While not directly related to A/B testing, they teach you to be metrics driven. This will help you setting up A/B tests as a PM.
@Felipe, Any recommendation or link to a good execution case study?
@AhmadBashir, any of the product execution YouTube videos or Medium blogs will do, since you just want to learn about product metrics to use for A/B testing.
Think of it like a science experiment. You have the control and the test group(s). Each of the test groups will be compared to the control and that helps say whether that individual experience is better than control and should be pushed to production / to all users.
Keep in mind there are lots of metrics you look at to determine if something should GA. It shouldn’t not only make sense holistically , but also should align with the product / company strategy.
FYI the bucket sizes should all be the same size. You will want a size that will give you a statistically significant results within a few weeks. If you have large audience, you need a smaller size. You will need either software or data scientist or you need to know a bit of statistics to say if the results are significant (ie not a change in metrics by chance)
To bounce off this question, are there any books on this topic?
I really like the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. It also has a chapter on organizational metrics, which is helpful for getting a better understanding to the why behind A/B testing and metrics in general.
I’ve been doing A/B testing for pretty much my whole career. Have done both in-house and at experimentation focused agencies. My focus has been mainly on the web but the foundation is still the same.
Ron Kohavi who has done testing at Airbnb, Microsoft, and Amazon has a great book caled Trustworthy Online Controlled Experiments
If you are a visual/auditory learner like myself, this video is extremely helpful for you to see the big picture as well as some specific examples of being a Growth PM, which is a term that some companies use these days.
Also as someone had mentioned, Georgi Georgiev is great and knows his stuff. He devised an agile calculator (cheap but not free) for testing that helps to make faster business decisions compared to other testing calculators out there. Not sure how much traffic your product gets but low traffic is always a hurdle for testing.
As others have also said, CXL is useful for quick learning and inspiration.
My tip is, don’t get caught up in the process too much. I made the mistake of spending too much time doing preparation for testing and not enough on actually getting tests out earlier in my career.
It’s important to have the knowledge but actually being effective in running tests is a whole another skillset. Testing velocity is arguably the most meaningful measure of an effective experimentation program. Your manager, his/her boss, and your peers will want to hear about the tests you ran, what you learned from it and what was gained from it. Of course many will want to see positive results on the KPIs but I’ve always emphasized the learning aspect more because the learnings can scale across the org. If you focus only on the key metrics being measured, you are also risking your performance to be reliant on your test wins, which will no doubt have its ups and downs.
Check out the articles, videos, and case studies produced by the SaaS tools that companies use to run these kind of tests like Optimizely, HotJar, MixPanel, Kissmetrics, or probably anyone on this grid. You could even request a demo from several of them and get advise from their sales engineers based on your exact needs. If your company already has one of these tools, then you might even have access to their customer success team to help you out.
Also highly recommend the CXL institute + follow a bunch of CROs like Craig Sullivan, Georgi Georgiev
Could hit any of the online learning websites and I imagine they have some nice courses, for example https://www.udemy.com/courses/search/?q=running%20A%2FB%20tests
If you have a website you can painlessly set tests up for free using google analytics. walkthrough here. Create an A/B test - Optimize Resource Hub .
Of course that won’t give you a dataset to analyze if you want to go through the stat side on the back-end, but should give a good idea of how to implement/use them.
I worked in A/B testing and related analytics for nearly 3 years before making the jump to PM.
Personally however, even when starting out, it seemed extremely simple. You have a hypothesis, control and then any number of variants. The statistical complications come in when you’re trying to do multivariate testing but if you’re doing multi-variant tests, it speaks to a lack of prioritization and work on hypothesis.
Start at Aristotle, Plato and Karl Popper. This is when ideas started to really be documented on the scientific method.
Then work your way up to randomized control trials from the medical community.
That way, you have a proper base and you will know how the “method” really works and why; falsification, testing and why you do randomization, what is statistical significance and how you can arrive to N for your hypothesis and product.
Keywords for YouTube videos: experimental design courses, research methods, hypothesis testing, falsification, randomized control trials, p-hacking (good to know this one), segmentation, statistical significance.
I would recommend reading Evan Milers series on AB testing - https://www.evanmiller.org
In addition to being a great resource for learning about AB testing - he also has put together a set of free tools for running ab tests that you can use to create examples / play with to learn
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.