5 posts tagged with "statsig"

The Experimentation Paradox refers to a core tension that I've noticed in A/B testing and conversion rate optimization (CRO). I haven't been able to find anything about it online, so the name is my own.

It suggests that the bigger the change you make to a website or app, the less you need an experimentation tool to know if it's significantly different from a control. On the flip side, the smaller the change, the more critical testing tools become—but it also gets harder to see statistically significant results.

Imagine you designed and manage the website for a small business, and your job is now to improve sales on that site. How do you improve your website's conversion rate or the percentage of people who actually end up buying something? A quick search will lead you down the rabbit hole of conversion rate optimization (CRO). You'll learn about A/B testing, experimentation, and hundreds of tools you can buy to simplify this daunting process. Unfortunately, most of these tools won't work for businesses with fewer than 100,000 visitors per month. What should you do?

Read on to find out how Topeca Coffee Roasters doubled their small eCommerce website's conversion rates.

Ezbot is using deep reinforcement learning to transform how user experiences are optimized completely. We will start with the basics of A/B test-based optimization and some problems that arise, like how slow, complex, and costly these programs can be. Then, we'll consider how deep reinforcement learning can solve these problems with higher sample efficiency and agentic decision-making. Finally, we'll follow data through ezbot's data collection, training, and prediction pipeline to explain how ezbot works behind the scenes.

A/B Testing: the Best Way to Improve Websites?​

A/B testing has been around for nearly 20 years now, and many people have built their careers on some form of it. Large experimentation teams at companies run multivariate (A/B/C/etc) tests and tease out any potential interaction effects between experiments. Still, most of the tests run don't yield any useful information. What are A/B tests and why do people swear by them? If experimentation is so powerful, why don't teams get more information? Isn't there a better way to improve websites?

In the relentless pursuit of conversion rate optimization (CRO), A/B testing has long been the reigning champion. It allows you to compare variations of your website elements, from headlines to layouts, and identify the statistically "better" performer. But for all its strengths, A/B testing comes with a soul-crushing enemy: the wait for statistical significance.