Episode 4: How to Build an Experimentation Machine and Where Most Go Wrong

High Signal

Ramesh Johari (Stanford, Uber, Airbnb, and more) explores the art and science of online experimentation, especially in the context of marketplaces and tech companies.

Ramesh shares insights on how organizations evolve from basic experimentation practices to becoming fast, adaptive, and self learning organizations. We dive into challenges like the risk aversion trap, the importance of learning from negative results, and how generative AI is reshaping the experimentation landscape.

We also talk about common failure modes and the types of things you're probably doing wrong, along with strategies to avoid these pitfalls. Plus, we discussed the role of incentives, the necessity of data driven decision making, and what it means to experiment in high stakes environments.