A/B Testing

The End of "I Think" and the Start of "I Know"

Written by Carrie Brill

If you’ve ever sat in a room where the loudest voice in the meeting (rather than the best idea) dictated the roadmap, you know why A/B testing is actually a survival skill. We’ve all been there: The CEO wants the button to be blue because "it pops." The designer wants it to be ghost-white because it’s "clean." As the PM, you’re stuck in the middle, trying to translate opinions into actual value. This is where A/B testing becomes your best friend. It’s the ultimate peacekeeper. It moves the conversation from "I feel like this will work" to "The data says this works."

What it actually is (without the jargon)

Strip away the technical definitions, and A/B testing is just a way to stop guessing. Instead of betting the entire house on a hunch, you place small, calculated bets. You show Version A to half your users and Version B to the other half. Then, you let the users tell you who’s right.

Bridging the communication gap between engineers and stakeholders

Why I love it (and why you should too)

As a Product Manager, you are constantly balancing risk. Stakeholders want big, flashy launches; engineers worry about stability. A/B testing bridges that gap.

  • It settles the "HiPPO" problem: When the Highest Paid Person’s Opinion dominates the room, data is your only defense.
  • It allows for safe failure: You aren’t rolling out a mistake to 100% of your user base. You’re testing it on 10%. If it tanks, you roll it back. No harm done.
  • It creates incremental wins: We often chase the "silver bullet" feature that changes everything. But in reality, product growth usually comes from stacking 1% wins on top of each other over time.

Keeping the process sane

You don’t need a PhD in statistics to do this well, but you do need discipline. Here is how I try to keep it simple:

  • Don't test everything: Pick a fight that matters. Are you trying to fix drop-offs at checkout? Increase signups? Focus on one metric.
  • Have a real hypothesis: "Let’s see what happens" isn’t a strategy. "I bet changing this copy will clarify value and increase clicks" is.
  • Run it, then step away: This is the hard part. You will want to peek at the results on Day 1. Don’t. It’s like baking; if you open the oven too early, the cake collapses. Wait for statistical significance.
Avoiding common communication pitfalls in product management

The Takeaway

A/B testing is a reminder that we aren't our users. What makes sense to us (or our stakeholders) might be confusing to them.
If you aren't doing this yet, start small. Test a headline. Test a CTA placement. It gives you the confidence to walk into that next stakeholder meeting and say, "We aren't guessing on this one. We know."

Ready to settle the debate? If you’re tired of roadmap decisions being made by whoever shouts the loudest, it’s time to let the data speak up. Whether you need the engineering muscle to build the test or the product strategy to interpret the results, we can help you turn "I think" into "We know."
Let’s Get Testing