Aleks unpacks how Typeform AI acts as a co-pilot that doubled activation and boosted one-day conversion, plus the design shifts (CTA altitude and onboarding) that increased adoption. She also reveals why they moved video features down-tier and what a head-to-head test showed: video interviewers generated 14x more words and 10x fewer skipped questions with comparable completion time.

Finally, Aleks breaks down the cultural side—eliminating “anti-knowledge,” standardizing experiment design, and creating a cross-functional review that prevents false learnings—along with how her data engineering team evaluates LLMs for quality, latency, and trust.

Timestamps
[00:45] – Rethinking experimentation: “earn the right to A/B test” with staged rigor
[03:31] – Pricing and packaging shift: from forms to flows, ASP up 32%, annual attach up 25%
[05:34] – Typeform AI as a co-pilot: doubling activation and lifting one-day conversion
[07:27] – Adoption lessons: elevating AI CTAs and reducing friction to use
[10:02] – Behind the scenes: model selection, quality bars, and why MVP can backfire in AI
[12:40] – Moving video down-tier: demand signals, cannibalization checks, and net gains
[14:52] – Video vs. standard forms: 14x more words, 10x fewer skips, similar completion time
[20:22] – Building an experimentation culture: process resistance, “anti-knowledge,” and cross-functional review
[30:16] – Leader playbook: visibility, empathy, and incentives for rigorous testing

Takeaways
- Implement a staged experimentation funnel—discovery, simulation, then customer A/B—to reduce risk.
- Use pricing experiments to trade volume for revenue quality; pair higher monthly prices with stronger annual discounts to grow annual attach.
- Treat AI as an activation lever: elevate AI-first CTAs and streamline onboarding to boost adoption.
- Add video interviewer options to increase response richness (14x more words) while keeping completion rates steady.
- Enforce experiment hygiene: change one variable at a time, randomize at the right unit (account vs. user), and run long enough for effect size.
- Purge “anti-knowledge” by standardizing design, instituting cross-functional reviews, and only codifying learnings supported by repeatable data.

Connect with the Guest
Aleks Bass LinkedIn: https://linkedin.com/in/aleksbass
About the guest: https://www.growthbook.io/podcast/guests/aleks-bass
Episode page: https://www.growthbook.io/podcast/episode/1-1
Company Website: https://www.typeform.com

Raj Mehta, Vice President of Product and Technology at Moxie Pest Control, outlines how he turned a spreadsheet-run operation into a data-driven engine across 9,000+ daily calls.

Raj shares how consolidating fragmented systems into a data lake unlocked automation—from shrinking lead routing from 20–25 minutes to under 30 seconds—to deploying AI-powered call intelligence that scores every sales and retention conversation against a playbook. He breaks down a practical roadmap for traditional businesses: build MVPs, pilot in one branch with a trained feedback team, iterate fast, then scale.

You’ll hear how he positioned AI as a growth amplifier (not a job cutter), the difference between deterministic automation and LLM use cases, and the measurable impact: a 5% lift in conversion that compounds in a recurring-revenue model. Plus, Raj’s concise advice for leaders bringing AI into operations without breaking trust or momentum.

Timestamps
[00:45] – Guest intro: Raj Mehta, Moxie’s tech transformation, and 9,000+ daily calls
[01:20] – Starting point: 90% of ops in spreadsheets; why a data lake became the foundation
[02:45] – Automating time-to-lead: 25 minutes to <30 seconds and a 5% conversion lift
[04:27] – Roadmap design: MVPs, single-branch pilots, and scaling what works
[06:05] – Culture building: framing AI as growth and upskilling, not headcount cuts
[07:34] – Two lanes of automation: deterministic scripts vs. LLM-driven workflows
[08:45] – Call intelligence: scoring every sales/retention call and coaching at scale
[14:05] – Impact and advice: recurring revenue compounding and Raj’s playbook for getting started

Takeaways
- Build a single source of truth (data lake) to power automation and AI reliably.
- Cut time-to-lead with workflow automation and track the downstream impact on conversion.
- Pilot in one branch with a trained “feedback team,” iterate, then roll out—don’t scale too soon.
- Position AI as a growth multiplier; retain and upskill top performers to shape the culture.
- Separate deterministic automation from LLM use cases; do deep discovery with frontline teams.
- Use AI call intelligence to score every call against your playbook, surface coaching themes, and save manager time.

Connect with the Guest
Raj Mehta LinkedIn: https://linkedin.com/in/raj-milan-mehta
About the guest: https://www.growthbook.io/podcast/guests/raj-mehta
Episode page: https://www.growthbook.io/podcast/episode/1-2
Company Website: https://www.moxieservices.com

Vinoj Kumar, Vice President of Engineering at Upwork, leads at the intersection of infrastructure and product, where feedback loops are longer and the blast radius is wider.

He shares a pragmatic framework for experimentation—blast radius x reversibility—that sets testing rigor, plus how he measures success in product terms: faster search, resilient marketplace trust, and developer velocity. Vinoj explains why “high engagement” can mask low-quality experiences, how his team instrumented an internal NL chatbot with turns-to-success and downstream signals (like fewer JIRA tickets), and how a composite metric—cost per quality inference (CPQI)—aligns finance, engineering, and data science by uniting cloud costs, performance, and model accuracy.

He details where AI is already paying off (build pipelines, incident detection, testing), how to monitor model drift post-launch, and why some wins on paper must be killed in production to protect trust—like a high-hit-rate caching project that surfaced stale profile data.

Expect concrete practices: shadow traffic, slow canaries, synthetic staging that mirrors reality, feature flags, LLMs-as-judges, and the mindset to tie infrastructure to business outcomes.

Timestamps
[00:45] – Guest intro: Infrastructure meets product—and why experimentation looks different
[01:36] – Deciding what to test: blast radius x reversibility; canaries, shadow traffic, ship-and-monitor
[03:09] – Defining “good”: internal dev metrics vs. marketplace outcomes—and when engagement lies
[06:23] – Case study: “Talk to Data” chatbot—thumbs, turns-to-success, and reduced JIRA tickets
[09:45] – CPQI: a composite metric for cost, performance, and model quality that breaks silos
[16:55] – AI in engineering: build-time gains, MTTR/MTTD, agentic testing, and drift monitoring
[24:06] – The caching miss: 92% hit rate, stale data, trust risks—and what to do instead
[29:12] – Career advice: balance stability with bold experiments; always link infra to business value

Takeaways
- Decide testing rigor with blast radius x reversibility; reserve heavy testing for irreversible, high-impact systems.
- Measure quality by efficiency and success ratio—not raw clicks or query counts.
- Instrument NL tools with “turns to success” and track downstream impact (e.g., fewer ad hoc data tickets).
- Build composite metrics (e.g., CPQI) to align finance, engineering, and data science around shared outcomes.
- Use AI to accelerate builds, detect incidents sooner, and evaluate models; watch MTTR and MTTD.
- Treat ML features as living systems: feature-flag rollouts, realistic staging, drift monitoring, and LLM-as-judge evaluations—and be willing to kill “wins” that erode trust.

Connect with the Guest
About the guest: https://www.growthbook.io/podcast/guests/vinoj-kumar
Episode page: https://www.growthbook.io/podcast/episode/1-3
Company Website: https://www.upwork.com

So how do you measure value, ship safely, and scale what works? Marco Casalaina, VP of Products, Core AI and AI Futurist at Microsoft, joins to unpack how his team builds and evaluates next‑gen AI—at hyperspeed.

Marco leads the AI Futures team and previously led Azure OpenAI, Azure Cognitive Services, Responsible AI, and AI Studio; before Microsoft, he ran Salesforce Einstein.

He explains why enterprise value is best measured by go‑lives and real usage, how Microsoft’s Foundry equips developers with agent‑specific evals (tool call accuracy, task adherence), and why old metrics like “accepted completions” don’t fit modern dev loops.

We dig into model routing now productized across model families, orchestration frameworks and the Copilot SDK, shared memory experiments, and the rise of self‑verifying agents that iterate to defined thresholds.

Expect concrete examples—from rewriting docs for coding agents to Ralph loops with browser testing—and practical advice for leaders: major in evals, set acceptable error rates by use case, and get hands‑on with the tools daily.

Timestamps
[00:45] – Guest intro and Microsoft’s enterprise AI focus
[02:07] – Measuring value: go‑lives, telemetry thresholds, and token volume
[03:43] – From chatbots to agents: Foundry evals (tool calls, task adherence) and A/B testing in Microsoft 365 Copilot
[06:17] – When “good” changes monthly: model routing productized across model families
[07:23] – Orchestration and Copilot SDK: agents that create their own tools; OpenClaw and shared memory experiments
[11:45] – Engagement redefined: coding agents read your docs; writing for agents vs. humans
[14:55] – New dev loops: why accepted completions died; Ralph loops and self‑verifying builds
[17:06] – Evals in practice and guardrails: thresholds, non‑determinism, and out‑of‑domain tests; how to keep up without burning out

Takeaways
- Measure value by go‑lives and real usage (token volume), not time in portals or playgrounds.
- Evolve evals for agents: track tool call accuracy/success and task completion/adherence; A/B test models and strategies.
- Productize adaptability with model routing to match tasks to the right model family as capabilities shift.
- Build self‑verification into workflows: pair agents with automated testing (e.g., browser runners) and iterate to thresholds, not perfection.
- Write for agents as readers: tighten documentation, ship vetted code samples, and monitor bot traffic patterns.
- Guardrail open‑world agents: add out‑of‑domain evals and explicit capability limits; set acceptable error rates based on the stakes of your use case.

Connect with the Guest
Marco Casalaina LinkedIn: https://www.linkedin.com/in/ACoAAAAZPaYBs2PGoy3GU2rZqfYPBgkHN68pUag
About the guest: https://www.growthbook.io/podcast/guests/marco-casalaina
Episode page: https://www.growthbook.io/podcast/episode/1-4
Company Website: https://www.microsoft.com

Timestamps
[00:02] – Truist overview and Charles’s mandate: improving developer experience at scale
[00:56] – AI as a strategic priority; shifting quality left with automation to remove bottlenecks
[02:19] – Measuring success: DORA metrics plus sentiment—eliminating toil to drive happiness
[04:33] – Human-in-the-loop AI for high-stakes finance; customer and internal use cases
[07:25] – How Truist evaluates tools: personas, pain points, and starting with tests, docs, security
[08:35] – The stack: Microsoft Copilot, GitLab’s AI gateway approach, and tracking utilization
[10:36] – New skills and culture: prompt engineering, “managing” AI agents, and strong governance
[20:45] – What’s next: personalization vs privacy, fintech agility + bank stability, and where AI pays off now

Takeaways
- Shift quality left with automated checks so developers catch issues early without human gatekeeping.
- Measure DORA metrics and developer sentiment; remove mundane toil to increase speed and satisfaction.
- Keep humans in the loop for AI-assisted coding and customer answers—trust but verify in regulated contexts.
- Build an AI ecosystem with clear purposes (productivity, engineering, consumer) and a steering committee to avoid duplication.
- Start with low-risk, high-yield AI use cases—unit tests, documentation, and security triage—to build confidence and momentum.
- Upskill teams in prompt engineering and AI oversight so developers can effectively direct and review AI “agents.”

Connect with the Guest
About the guest: https://www.growthbook.io/podcast/guests/charles-williams
Episode page: https://www.growthbook.io/podcast/episode/1-5
Company Website: https://www.truist.com

Timestamps
[00:45] – Makram’s path: running experimentation at LinkedIn and Intuit, and why nothing ships without an A/B test
[02:15] – Costly lessons: 5px banner change, algorithm tweaks, and the case for rigorous guardrails
[06:40] – Leadership discipline: killing features (voice meetups, LinkedIn Stories) and changing OKRs to reward learning
[11:05] – People, process, technology: top-down and bottom-up tracks, and embedding “fail forward”
[13:40] – From MVP to MVT: validating leap-of-faith assumptions, PRFAQ, and rapid “unit of one” prototypes
[15:55] – Bottlenecks and unlocks: engineering/data science capacity, centers of excellence, and AI for fast front-end tests
[22:45] – Personalization at the top of funnel: avoid waste, design reviews, and right-size testing before building
[25:45] – Guardrail metrics that matter: AI infra costs, ethics/compliance, and fairness-by-design
[29:45] – ID.me now: zero-to-one builds, vision-to-values, North Star and leading indicators
[33:30] – How to start at a new org: crawl-walk-run, small wins, relationships, and over-communication

Takeaways
- Shift from MVP to MVT: list leap-of-faith assumptions and design minimum viable tests before you build.
- Institutionalize learning: align OKRs with “fail forward,” and be willing to kill low-performing features quickly.
- Build the triad: pair an easy-to-use platform with training, top-down sponsorship, and clear launch processes.
- Add real guardrails: track AI infrastructure costs, ethics/compliance, and inclusion metrics alongside growth KPIs.
- Unblock teams: create a center of excellence for data science and enable rapid variants with AI-powered tooling.
- Start small and visible: rack up quick wins, over-communicate progress, and grow influence through relationships.

Connect with the Guest
Makram Mansour LinkedIn: https://www.linkedin.com/in/ilmansour
About the guest: https://www.growthbook.io/podcast/guests/makram-mansour
Episode page: https://www.growthbook.io/podcast/episode/1-6
Company Website: https://www.id.me

Timestamps
[00:34] – Startup A/B testing mindset: de-risking big bets with only a 25% win rate
[02:45] – When to A/B test vs. ship: risk appetite, funnel stage, and non-inferiority tests
[04:43] – 360 experiments with 4 people: scaling to 1,000 using AI and Cursor cloud agents
[08:22] – Separating feature impact from momentum: PLG and trial model moves ARR
[10:29] – Ship PLG features, then iterate to drive usage; measuring DAU and revenue impact
[11:40] – Habit loops to growth loops: turning product features into PLG (scheduling case study)
[16:47] – Building an experimentation culture: founder buy-in, Slack changelog, shared data
[26:50] – The modern growth stack: Claude, Cursor, GrowthBook, BigQuery, and DOT in Slack

Takeaways
- Prioritize by risk: run rigorous A/B tests where you have volume; use before/after or non-inferiority for low-risk in-product changes.
- Test big levers—not just UI: pricing models, usage limits, onboarding pathways—and judge success by ARR movement, not micro-metrics.
- Ship first, then optimize: launch PLG features and immediately run experiments to increase adoption; track daily active usage per feature.
- Build growth loops from habits: design shareable artifacts and personalized signup paths; drive users back to your domain to capture value.
- Scale experimentation with AI: use Cursor desktop/cloud agents for parallel builds and visual QA; orchestrate docs/analysis via Claude; automate cleanups and reporting.
- Make experimentation company-wide: centralize data (BigQuery), broadcast wins/losses in Slack via GrowthBook, and auto-correlate metric dips to releases.

Connect with the Guest
Kameron Tanseli LinkedIn: https://www.linkedin.com/in/kameron-tanseli/
About the guest: https://www.growthbook.io/podcast/guests/kameron-tanseli
Episode page: https://www.growthbook.io/podcast/episode/1-7
Company Website: https://www.fyxer.com

Timestamps
[00:45] – Ancestry’s experimentation maturity: metering, paywalls, and the 5.3% lift that unlocked capabilities
[03:48] – From isolated tests to a capability mindset: experimentation as an operating system
[05:34] – Balancing wins with learning: zooming out for subscription engagement and NPS
[08:49] – Operating model: centralized platform, self-serve dashboards, and baseline resets
[11:59] – Counterintuitive UX lesson: removing friction backfired (–20–25% CTR); respect mental models
[15:10] – AI storytelling as a growth lever: record comparisons into narratives, 30%+ CTR and 5x views
[18:27] – Input vs. output metrics: when to roll back and how to link short- and long-term outcomes
[30:00] – Parting advice: test what changes CX, avoid vanity testing, and optimize decision quality

Takeaways
- Build capabilities, not just tests—use experiments to unlock platform features (e.g., metering, paywalls).
- Democratize experimentation with a centralized platform and self-serve tooling; reset baselines regularly.
- Test “obvious” UX changes; preserve helpful friction and align with user mental models.
- Turn data into narratives with AI to deepen engagement and increase discovery.
- Define input and output metrics; ship only what improves core outcomes (retention, sign-ups), and roll back fast if not.
- Optimize for decision quality: right audience, sufficient sample sizes, clean baselines, and true statistical significance.

Connect with the Guest
Suresh Teckchandani LinkedIn: https://www.linkedin.com/in/teckchandani/
About the guest: https://www.growthbook.io/podcast/guests/suresh-teckchandani
Episode page: https://www.growthbook.io/podcast/episode/1-8
Company Website: https://www.ancestry.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

In this episode, Ashley Stirrup sits down with Nafis Shaikh, Director of Product Management at Chess.com, to get inside the experimentation engine powering one of the world's most beloved gaming products.

Nafis brings experience from Zynga and Prodigy and a refreshingly honest take on what changes when a product built on passion suddenly has to serve a 10-million-DAU user base that spans absolute beginners to rated FIDE players. He and Ashley get into why one-size-fits-all doesn't actually fit anyone, how to measure an AI coach when you can't tell whether users have their volume on, and a game review experiment that completely upended the team's assumptions about how players want to learn.

Nafis also shares practical advice for product managers trying to introduce experimentation culture to organizations that have never done it before—starting with a simple pre/post test rather than a fancy platform. If you lead product, care about experimentation maturity, or just want to hear how a classic product is scaling its learning loop, this one's worth your time.

Timestamps
- [00:35] – Chess.com's experimentation origin story and the 1,000-test goal
- [05:01] – Designing for a user base that spans beginners to FIDE-rated players
- [07:30] – The four metrics dimensions Nafis uses to evaluate tests
- [12:03] – How do you A/B test an AI coach when you can't tell who's listening?
- [15:49] – Embracing humility and the shift away from "we know what works"
- [20:50] – The game review test that surprised everyone: 80% of users review wins
- [24:06] – Advice for PMs introducing experimentation at a new company
- [29:15] – The onboarding debate and personalization from session zero

Takeaways
- Scale test volume to learning speed, not just shipping speed
- Build hypotheses around user psychology, not just KPI movement
- Accept that being wrong is the point—experimentation only works when leadership embraces humility
- Start simple if you're new to experimentation; a clean pre/post comparison beats a fancy platform you don't use
- Reposition features around how users actually feel, not how you assume they should feel
- Design onboarding around the shortest path to value, not the longest path to personalization

Connect with the Guest
Nafis Shaikh LinkedIn: https://www.linkedin.com/in/nafis-shaikh-20161916/
About the guest: https://www.growthbook.io/podcast/guests/nafis-shaikh
Episode page: https://www.growthbook.io/podcast/episode/1-9
Company Website: https://www.chess.com

The difference isn't tooling or headcount. It's culture. When your CEO reads Excel spreadsheets for fun and actively wants data to prove him wrong, you stop debating whether to test and start debating how to test smarter. Medha shares the frameworks Fanatics uses to balance speed with rigor: a "do no harm" track for brand plays that won't show up in conversion metrics, a small-sample framework for teams that can't hit statistical significance thresholds, and an experimentation Wiki that feeds a continuous iteration flywheel. One surprising test on ad removal initially showed 95% statistical significance—until they replicated it and found the result was a false positive. The lesson: even at scale, you need to double-click on causality.

Timestamps
03:09 How Fanatics scaled from 10 to 100 experiments per month over 10 years
05:25 Why some leadership teams embrace experimentation and others resist it
07:06 How experimentation consistently delivers a big portion of Fanatics' annual growth
08:20 What happens when your CEO consumes Excel spreadsheets and questions everything
10:35 How top-down humility shapes an entire company's testing culture
12:10 The ad removal test that looked like a 95% win—then failed replication
15:55 How Fanatics built an experimentation Wiki that powers their growth engine
22:45 The "do no harm" framework for features that don't measure cleanly in A/B tests
25:20 Why lowering barriers to adoption matters more than statistical perfection early on
26:27 Your odds of winning at experimentation are worse than roulette

Takeaways
- Replication catches false positives: A 95% confidence level still means 1 in 20 results are noise—if a critical test outcome can't be explained through micro-metrics, run it again before committing resources.
- Top-down buy-in shifts the conversation from "why test?" to "how do we test?": When leadership treats data as the tiebreaker, teams stop defending opinions and start building better experiments.
- Frameworks like "do no harm" and "small sample" expand who can test: Not every initiative needs 30,000 orders to ship value—lower the barrier for teams that can't hit statistical thresholds while protecting core KPIs.
- Documenting experiments in a centralized Wiki creates a growth flywheel: Fanatics' Wiki feeds their roadmap with iterations on already-built features, reducing tech dependency and accelerating velocity.
- Micro-metrics establish causality beyond top-line KPIs: If revenue moves but scroll depth, cart adds, and product views don't follow the same pattern, question the result before declaring a win.

Connect with the Guest
Medha Umarji LinkedIn: https://www.linkedin.com/in/medhaumarji/
About the guest: https://www.growthbook.io/podcast/guests/medha-umarji
Episode page: https://www.growthbook.io/podcast/episode/1-10
Learn more about Fanatics
https://www.fanatics.com/

Summary
Dave Massey walked into UPS in 2016 and immediately got pulled into a meeting about AB testing tools. By the end of the day, he owned the platform—and the problem: UPS hadn't run a single meaningful experiment. Three years later, senior leadership gave him a hard number to hit. Prove UX could move revenue, or the pilot dies. His first test—removing navigation from the checkout flow—delivered $35 million in incremental revenue. Senior leaders didn't believe it. They made him defend the results upside down and sideways. When the dust settled, the data held. Today, Massey's team has driven over half a billion dollars in incremental revenue by treating UPS.com like the e-commerce business it actually is.

Massey's approach is simple: test everything, especially what senior leaders think will work. His team, Journey Experience and Design Innovation (nicknamed J.E.D.I.), has built a reputation for saying no with data, not opinion. When a business unit demanded required recipient emails to capture customer data, J.E.D.I. ran the test in 24 hours and killed it. Conversion tanked. Two years later, the international team asked for the same feature—but framed it as a customs solution. That test passed. Same feature, different reason, different outcome. That's the edge Massey's team delivers: rigorous hypothesis design, a UX research team embedded in the experimentation workflow, and zero tolerance for untested ideas.

Timestamps
03:09 Dave's first day at UPS: inheriting an AB testing tool with no program
05:59 Senior leadership's ultimatum: prove UX ROI or kill the pilot
08:38 First test result: $35M from removing navigation in checkout
09:48 Defending the numbers: how Massey's team survived scrutiny
11:07 Why a data-driven engineering culture made experimentation inevitable
16:12 Team size: 80 people supporting almost 80 customer-facing applications
19:08 The 24-hour test: when required email fields killed conversion
22:28 Why Massey embeds UX research inside the experimentation team
24:41 AI at UPS: treating it as a tool, not a replacement

Takeaways
- - Massey's first test removed navigation from UPS's shipping checkout flow and delivered $35 million in incremental revenue—proving e-commerce best practices apply even when customers think "this is just a tool, not e-commerce."
- J.E.D.I.'s win rate stays high because UX research and experimentation teams operate under the same leader, giving the program both behavioral metrics and voice-of-customer insight before tests ever launch.
- When senior leaders push ideas, Massey's team tests them instead of arguing—then delivers results that either validate the idea or identify three better alternatives the data actually supports.
- The same feature (required recipient email) failed for customer data capture but passed for international customs—proof that framing and customer benefit matter more than the feature itself.
- UPS runs everything centrally now, but the real win is that demand for testing has decentralized—business units across the company now come to J.E.D.I. asking to test their ideas.

Connect with the Guest
Dave Massey LinkedIn: https://www.linkedin.com/in/masseycreates/
About the guest: https://www.growthbook.io/podcast/guests/dave-massey
Episode page: https://www.growthbook.io/podcast/episode/1-11
Learn more about UPS: https://www.ups.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Topics: A/B testing, experimentation, conversion rate optimization, feature flags, UX research, e-commerce experimentation, statistical significance, experimentation team building, growth experimentation, sequential testing.

Summary
Most companies struggle to scale experimentation beyond engineering teams. DoorDash runs over 12,000 experiments per year across 42 million monthly active users — and now they're enabling restaurant owners to run their own A/B tests on menu pricing and promotions. Ilya Izrailevsky, Senior Engineering Manager leading DoorDash's experimentation platform, shares how the company built a three-sided marketplace testing program that balances consumers, dashers, and merchants across 40+ countries. From his time scaling search at Amazon (where offline model evaluation narrowed hundreds of candidates down to 10 for live testing) to preventing DashPass churn at DoorDash, Ilya reveals what happens when experimentation scales beyond product teams — and why CEO-level experiment review emails drive cultural change faster than any training program.

One standout learning: expanding delivery radius to 11+ miles increased grocery orders but tanked retail conversions. The lesson wasn't about distance — it was that one metric approach breaks in multi-dimensional marketplaces. DoorDash now segments experimentation by vertical, behavior pattern, and regional market, using AI agents to mine institutional knowledge from past tests and auto-generate experiment summaries that ship company-wide within hours of readout.

Timestamps
00:40 From building Wasabi (Intuit's open-source platform) to running ML at Amazon and Uber
03:04 Why product velocity without experimentation creates feature bloat, not impact
05:32 Scaling search at Amazon: billions of products, 10 visible results, 25% win rate
08:22 Offline evaluation as a filter — golden data sets cut model candidates before live traffic
10:23 DoorDash's three-sided marketplace: 300 million feature flag evaluations per second
12:38 CEO Tony Xu reads every experiment email and replies with alternative hypotheses
13:33 Democratization at scale: enabling merchants to A/B test menu pricing and promotions
17:05 DashPass churn experiment uncovered value perception gap — became a full product area
22:03 Why expanding delivery radius killed retail orders but boosted grocery conversions
24:16 No single North Star metric — balancing consumer quality, dasher earnings, merchant mix
27:29 Four-dimensional scale: democratization, global expansion, new verticals, AI agents
31:03 Agentic experimentation: AI mines past tests to generate hypotheses and debug imbalance

Takeaways
- Win rate matters less than learnings per test — DoorDash ships company-wide experiment summaries (win or lose) that the CEO actively reads and responds to, creating cultural accountability around testing rigor.
- Offline evaluation acts as a pre-filter for model velocity — Amazon's search team used golden data sets to cut hundreds of ML candidates down to 10 for live A/B testing, preventing wasted experiment slots.
- One-size metrics break in multi-dimensional marketplaces — DoorDash balances consumer retention, dasher utilization, and merchant inventory mix across verticals because optimizing one side degrades the ecosystem.
- Democratization requires opinionated templates, not open-ended tools — enabling non-technical users to run tests means embedding success metrics and guardrails into pre-built experiment configs.
- AI scales institutional knowledge, not just analysis speed — mining past experiment readouts to auto-generate new hypotheses turns your testing history into a compounding advantage.

Connect with the Guest
Ilya Izrailevsky LinkedIn: https://www.linkedin.com/in/ilyaizrailevsky/
About the guest: https://www.growthbook.io/podcast/guests/ilya-izrailevsky
Episode page: https://www.growthbook.io/podcast/episode/1-12
Learn more about DoorDash: https://www.doordash.com/

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Topics: A/B testing, experimentation platform, feature flags, marketplace experimentation, machine learning, growth experimentation, statistical significance, experimentation culture, agentic AI workflows.

Summary
Andrew Willingham, Head of Legal and People Products at Atlassian, spent 11 years at Amazon before joining Atlassian a year ago. His path from running A/B tests on millions of Amazon shoppers to building talent management software for a few hundred thousand employees forced a fundamental shift: when you can't run tests at scale, you have to sit with your actual users and watch them fail. He shares how building a talent review product for Amazon's HR specialists completely flopped when handed to HRBPs — and why that failure taught him more than any winning experiment. Now at Atlassian, he's applying that same rigor to reimagining hiring processes with AI, testing everything from recruiter screens to interview sequences that the industry has run the same way for decades.

Timestamps
03:09 From marketing Amazon's mobile app to building HR software for 1.5 million associates
08:19 Why a talent review product loved by IO psych experts flopped with actual HRBPs
11:11 How A/B testing helps product managers escape opinion-based politics
15:25 Testing copy that changes behavior: "We'll generate that status report for you"
17:20 The two North Star metrics Andrew optimizes: efficiency and quality
19:05 Khan Academy's metric trap: measuring cognitive engagement, not just completion
21:10 Why product managers resist experimentation — and what changes when you admit you don't know

Takeaways
- Your customer and your user may not be the same person — building for HR specialists instead of the HRBPs who actually run talent reviews resulted in a feature nobody could use.
- When you can't test at scale, desk rides replace A/B tests — sitting with users and watching them struggle reveals failures faster than any dashboard.
- Experimentation short-circuits political debates by removing opinion from product decisions.
- Test metrics before you test features — usage time could signal engagement or just mean your product takes too long to do its job.
- The experiments that fail deliver the most valuable learnings, especially when you expected a slam dunk.

Connect with the Guest
Andrew Willingham LinkedIn: https://www.linkedin.com/in/andrewwillingham/
About the guest: https://www.growthbook.io/podcast/guests/andrew-willingham
Episode page: https://www.growthbook.io/podcast/episode/1-13
Learn more about Atlassian: https://www.atlassian.com/

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Topics: A/B testing, product experimentation, feature flags, user research, talent management, qualitative research, metric design, experimentation at scale, growth experimentation.

Chapters
00:00 Intro
01:15 Geoffrey's path: Lowe's, Microsoft, Ford Credit
07:15 How Ford Credit fits with Ford Motor
10:15 The teams behind every Ford Credit page
15:15 The vehicle selector test that lost on purpose
19:15 Why feature placement beats feature ideas
21:15 Personalization and the shrinking-audience problem
25:15 Telling the story when a test loses
30:45 Connecting an online test to an offline car sale
33:55 The experimentation piggy bank

Takeaways
- 1. Losing tests often create more value than winners because they stop expensive mistakes before they ship.
- 2. Measure experimentation two ways: the revenue you earn from wins and the revenue you save by killing bad experiences.
- 3. A feature that fails early in a flow can succeed later; placement and timing often matter more than the idea itself.
- 4. Connecting online experiments to offline outcomes like receivables turns a small lift into a number leadership cares about.
- 5. When you struggle to land a result, lead with the story of what the customer did, then bring the numbers.

Connect with the Guest
Geoffrey Bell LinkedIn: https://www.linkedin.com/in/geoffrey-bell-62a03617/
About the guest: https://www.growthbook.io/podcast/guests/geoffrey-bell
Episode page: https://www.growthbook.io/podcast/episode/1-14
Company Website: https://www.ford.com/finance/

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

• (00:00) - Intro
• (01:15) - Geoffrey's path: Lowe's, Microsoft, Ford Credit
• (07:15) - How Ford Credit fits with Ford Motor
• (10:15) - The teams behind every Ford Credit page
• (15:15) - The vehicle selector test that lost on purpose
• (19:15) - Why feature placement beats feature ideas
• (21:15) - Personalization and the shrinking-audience problem
• (25:15) - Telling the story when a test loses
• (30:45) - Connecting an online test to an offline car sale
• (33:55) - The experimentation piggy bank

Chapters
00:00 Welcome and Mayank's path from quant trading to RingCentral AI
02:45 Why experimentation has to be owned cross-functionally
04:55 Small experiments that compounded to a 12% lift at Groupon
06:45 Why accuracy and thumbs-up/down fail for AI agents
08:15 The DART framework, metric by metric
12:45 Applying DART to AI-generated smart notes
14:55 The Groupon flash-sale that dropped conversion
16:45 Swapping price urgency for scarcity and social proof
19:45 North Star metrics, guardrails, and Goodhart's law
26:45 The future: experimenting on — and for — AI agents

Takeaways
- Accuracy is a comfortable lie. It grades a narrow test set and can stay high while the agent fails real users.
- Thumbs-up/down feedback is sparse and skewed. Unhappy users rarely rate — they just quietly stop using the product.
- DART measures behavior, not opinions. Four signals read off logs and transcripts: decay, acceptance, relevance, and task completion.
- Acceptance rate is the trust metric. The share of output users keep without editing is the strongest available proxy for trust.
- A losing experiment is paid-for information. Groupon's flash-sale flop revealed the lever was wrong, not the goal — scarcity beat price-based urgency.

Connect with the Guest
Mayank Agarwal LinkedIn: https://www.linkedin.com/in/mayank-agarwal-6223b04a/
About the guest: https://www.growthbook.io/podcast/guests/mayank-agarwal
Episode page: https://www.growthbook.io/podcast/episode/1-15
Company Website: https://www.ringcentral.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:45 What Signet Jewelers actually is (Kay, Jared, Zales, and more)
01:45 From art school to UX to experimentation: Craig's background
04:45 How experimentation is organized: a centralized model across brands
06:45 From 40–50 tests a quarter to 15–25 value-driven experiments
08:45 The "view all" experiment: adding friction to grow revenue
12:45 One product page, many stakeholders: financing, warranties, chat
15:45 Extracting learnings when an experiment loses
18:45 Why revenue per visitor beats conversion as the north star
20:15 Intent-based personalization and "engagement season is every day"
24:45 Bringing the whole org along by tying insights to dollars.

Takeaways
- Friction can increase revenue. Blocking the "view all" grid and forcing a style choice sent shoppers deeper and lifted conversion and revenue, because the extra click added value.
- The three-click rule is conditional. Clicks only hurt when they're empty; a click that narrows thousands of options to dozens is a feature, not a cost.
- Revenue per visitor is the honest north star. Conversion rate can be gamed to 100% by making everything free or cutting bounce-heavy traffic; revenue per visitor can't.
- Fewer, bigger experiments beat high volume. Signet went from 40–50 tests a quarter to 15–25 because complex, value-driven tests produce reusable insights that small tweaks don't.
- Tie every result to dollars. Translating experiment outcomes into revenue is how Craig keeps financing, warranty, and chat stakeholders aligned and gets executives to act.

Connect with the Guest
Craig Kistler LinkedIn: https://www.linkedin.com/in/craigkistler/
About the guest: https://www.growthbook.io/podcast/guests/craig-kistler
Episode page: https://www.growthbook.io/podcast/episode/1-16
Company Website: https://www.signetjewelers.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters 00:00 Introduction: Lina Blackman, Director of Product Analytics at Squarespace 01:45 Squarespace's business and 3 million website customers 02:30 Decentralized analysts, centralized experimentation program 04:15 150–200 experiments a year: onboarding, mobile, checkout, pricing 04:55 The blank template disaster that became Blueprint AI 07:45 Two questions for every losing test 09:30 Moving ship-first teams up the experimentation maturity curve 12:30 A/B test logs and insights rituals 13:30 North Star metrics and the KPI tree 16:35 AI in the A/B testing workflow — and what stays manual.

Takeaways

• Stated preference lies: users asked for a blank canvas, but behavior demanded guided design — and only the experiment could referee.
• Close every losing test with two questions: did it work for a granular segment, and is the idea worth further investment?
• One or two big wins a quarter is a healthy hit rate when you run 150–200 experiments a year.
• Calibrate certainty to stakes — tight bounds on revenue and pricing tests, wider bounds on engagement tests so teams don't spin on noise.
• Hand AI the mundane parts of the workflow (tracking, assignment setup), but if AI runs the brief and the analysis, ask why you're running the test at all.

  
  

Connect with the Guest
Lina Blackman LinkedIn: https://www.linkedin.com/in/linanguyen
About the guest: https://www.growthbook.io/podcast/guests/lina-blackman
Episode page: https://www.growthbook.io/podcast/episode/1-17
Company Website: https://www.squarespace.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:00 Welcome and introduction
01:15 Arun's background and marketing experimentation at Schwab
04:15 Uber's mature, experiment-driven culture
06:30 Coming to Twitch: from Python notebooks to a shared standard
08:30 The pricing problem Twitch had long avoided
10:30 Geo-fenced experiments, matched markets, and elasticity
13:15 The gifted-subs surprise and testing promotions
16:15 The discipline that matters before you push play
18:15 Why false negatives are worse than false positives
20:05 Enrollment triggers and broad explore experiments
22:45 AI, the Kiro tool, and what's next for experimentation

Takeaways
- False negatives are more dangerous than false positives — they get institutionalized as "we tried that, it didn't work" and quietly kill good ideas for years.
- The most valuable experiment work happens before you push play: clear enrollment logic, a plain-English hypothesis, and no optimizing ahead of the test.
- If an intervention sounds weak when you write it out in plain English, don't run the experiment — you're just wasting time.
- Run a broad explore experiment first; small, over-narrowed populations lack power and raise the odds of a false negative. Find the responsive segment with heterogeneous treatment effects afterward.
- Twitch used geo-fenced experiments with matched markets and causal inference to measure true price elasticity, turning a feared pricing decision into a measured, accretive one.

Connect with the Guest
Arun Bodapati LinkedIn: https://www.linkedin.com/in/abodapati/
About the guest: https://www.growthbook.io/podcast/guests/arun-bodapati
Episode page: https://www.growthbook.io/podcast/episode/1-18
Company Website: https://www.twitch.tv

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:00 Welcome to the experimentation edge
01:45 Kevin's role leading experimentation at chase
04:15 Why chase invested in experimentation
06:45 A billion dollars and the value of losers
12:45 Plan for failure to beat confirmation bias
14:30 The million dollar change you can't see
18:45 Sharing learnings and experimentation wrapped
20:45 Engagement without vanity metrics
22:00 Experimentation's golden era with AI
23:30 Why AI needs more experimentation, not less

Takeaways
- Chase estimates over a billion dollars of value from experimentation, and most of the lasting learning comes from the losing tests, not the winners.
- A control group is non-negotiable: at scale, a change worth millions is invisible under noise and seasonality, and no one can spot it by eye.
- Treat engagement carefully. For a bank, more time in the app isn't a win; trust, fast task completion, and healthy repeat engagement are.
- Plan for failure before you run a test. A pre-built playbook for a loss prevents confirmation bias and keeps teams from gaming the metrics.
- AI is ushering in a golden era for experimentation, because shipping faster only compounds mistakes unless you measure what you ship.

Connect with the Guest
Kevin Yang LinkedIn: https://www.linkedin.com/in/kevintyang
About the guest: https://www.growthbook.io/podcast/guests/kevin-yang
Episode page: https://www.growthbook.io/podcast/episode/1-19
Company Website: https://www.jpmorganchase.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:00 Intro
00:45 The mindset shift from shipping to results
02:00 Why testing took six months, a tooling problem
03:15 The dev team that laughed, and the vendor who agreed
04:50 An executive demand for video chat
05:35 Dry testing with a fake button
06:30 106 clicks and a multimillion dollar save
07:30 Savings and gains, not wins and losses
08:45 The Disney team that didn't know where to start
10:30 From low engagement to 110 prioritized ideas
12:45 Just get something live, and where AI fits next

Takeaways
- A "dry test", a fake "Click here to video chat" button that grayed out on click — measured real demand without building the feature. Of roughly 4 million users, only 106 clicked, killing a multimillion dollar build.
- Reframe experiment outcomes as savings and gains, not wins and losses. A "losing" test saves you from a costly mistake, which keeps teams focused on learning instead of fearing failure.
- Slow experimentation is often a tooling problem, not a methodology problem. One program's six month test cycle came from rebuilding every page instead of overlaying changes the way the tool intended.
- Getting a stuck team unstuck starts with data and a workshop. A Disney team went from "we don't know where to start" to 110 scored, prioritized test ideas, using Contentsquare heatmaps to diagnose low engagement first.
- The biggest thing that gets a team testing is to just do it. Stop designing the perfect experiment and get something simple live to take away the mystery.

Connect with the Guest
Crystal Ammari LinkedIn: https://www.linkedin.com/in/crystal-ammari/
About the guest: https://www.growthbook.io/podcast/guests/crystal-ammari
Episode page: https://www.growthbook.io/podcast/episode/1-20
Company Website: https://thewaltdisneycompany.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:00 Intro
00:45 From neuroscience research to Home Depot
01:45 A $150B enterprise, a $25B online business
02:45 The centralized experimentation model
03:45 Inside the 40-person team
04:30 Readouts, blast emails, and the experiment library
05:40 Executive visibility and the golden rule
06:15 "If you won't act on a bad result, don't run the test"
11:15 Learning from losing tests
12:30 Scaling up: AI, server-side testing, and what's next

Takeaways
- One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.
- Executive engagement is real at Home Depot: leaders join 30-minute readouts, search the experiment library, and ping analysts directly because they treat A/B testing as the golden rule for measuring incrementality.
- Institutional memory is infrastructure — every test result since 2020 lives in a centralized, searchable archive so no one re-runs a question the company already answered.
- Kim's stakeholder filter: if you wouldn't do anything differently after a bad result, don't run the test.
- Scaling past low hundreds of experiments per year is a capabilities problem before it's an AI problem — Home Depot is moving from client-side to server-side testing so winners release quickly, end to end.

Connect with the Guest
Kim Ting Li LinkedIn: https://www.linkedin.com/in/kimtingli
About the guest: https://www.growthbook.io/podcast/guests/kim-ting-li
Episode page: https://www.growthbook.io/podcast/episode/1-21
Company Website: https://www.homedepot.com

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:45 Welcome and what the show is about
01:45 Raunak's role and 12 years in marketing analytics
02:45 How AI and Claude Code changed the analyst's day
04:15 LLMs, declining organic traffic, and the 5x ChatGPT jump
05:15 Two kinds of experiments at Fin: on page and off page
07:15 The Jira Service Desk bundling experiment
10:45 Why the trial winner became a rollback
11:45 Contextual onboarding turns the loser into a winner
14:45 Reading an experiment that loses
18:45 What's next: incrementality, connected TV, and testing creative

Takeaways
- AI has collapsed marketing analysis from weeks to hours, and the real payoff is a cleared experiment backlog plus analysts who compete on the questions they ask, not the speed they query.
- Organic search traffic is declining as ChatGPT, Gemini's AI mode, and Claude answer buyers in place; Fin saw a 5x rise in ChatGPT referrals, but LLMs don't tag that traffic, so attribution has to be proven through experiments.
- A guardrail metric saved Atlassian from a costly mistake: bundling Jira Service Desk lifted trials more than 50 percent but tanked activation and paid conversion, forcing a rollback.
- A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.
- In low-volume B2B, read losing experiments for sub-segment signal; a "failed" Stripe form simplification revealed the form was blocking legitimate small-business buyers using Gmail.

Connect with the Guest
Raunak Kumar LinkedIn: http://linkedin.com/in/raunakkumar1991
About the guest: https://www.growthbook.io/podcast/guests/raunak-kumar
Episode page: https://www.growthbook.io/podcast/episode/1-22
Company Website: https://fin.ai

Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.

Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.

With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.

See a demo at https://www.growthbook.io/

Chapters
00:00 Cold open
00:50 Meet Amir and the air booking platform at Expedia
03:10 DoorDash, growth, and a 70 experiment year
04:20 Three kinds of experimentation at Expedia
06:30 AI velocity and the new frontier model pace
08:30 What you cannot measure, you cannot ship
10:45 The DoorDash carousel and the price experiment
12:45 The three sided marketplace and competing metrics
16:55 There are no losing experiments
20:45 Predictability, LLMs, and Expedia's road ahead

Takeaways
- "What you cannot measure, you cannot ship" — if you can't measure an outcome, you can't decide whether it's better, so you're just debating opinions.
- Measurement spans three live dimensions: spend (more with less), speed (sprints instead of quarters), and quality, with guardrail "do no harm" metrics on top.
- There are no losing experiments. A flat result is a signal to either refine the hypothesis or step back and look from a completely different angle.
- DoorDash's price experiment proved price by itself doesn't predict orders. Different customers want different things at different times, which pushed the team toward personalization.
- A three sided marketplace (buyers, merchants, Dashers) makes metrics compete. Running the test is easy; deciding what to optimize when goals conflict is the real work.

Connect with the Guest
Amir Moghaddam LinkedIn: https://www.linkedin.com/in/amirmoghaddam
About the guest: https://www.growthbook.io/podcast/guests/amir-moghaddam
Episode page: https://www.growthbook.io/podcast/episode/1-23
Company Website: https://www.expediagroup.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

Chapters
00:00 Cold open and welcome to the show
01:45 What Stitch Fix actually does
04:15 Balancing AI with the human stylist
05:15 From public policy to the A/B testing adrenaline rush
07:15 Inside the weekly experimentation review group
08:45 The AI style assistant and listening to qualitative feedback
10:45 Why adoption friction beats product bugs
13:45 Testing for losers and building guardrails
15:45 Keep rate, successful fixes, and the holy grail metric
18:15 The new platform and the promise of agentic AI

Takeaways
- The most impactful experiment learnings usually come from adoption friction, not product bugs. By the time a big feature reaches A/B testing, it's often already a winner, so the open question is how and where to introduce it.
- A losing test is a finding, not a failure. If every experiment wins, you're not taking enough risk to learn anything new.
- Guardrails and stopping criteria are what make risk-taking safe, especially when the experience is as personal as shopping.
- The most valuable North Star metric is the one you can't measure yet, long-term client value, and causal-inference modeling helps predict it from short-term behavior.
- Quantitative results are only half the story. Direct, qualitative client feedback inside an experiment often reshapes the rollout more than the numbers do.

Connect with the Guest
Nick Beyler LinkedIn: https://www.linkedin.com/in/nick-beyler-381864119/
About the guest: https://www.growthbook.io/podcast/guests/nick-beyler
Episode page: https://www.growthbook.io/podcast/episode/1-24
Company Website: https://www.stitchfix.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:45 Meet Dan Layfield and Diligent
01:45 Two worlds of experimentation, Codecademy and Uber
03:45 The trial model that lifted conversion 35%
06:20 What to do with a losing experiment
08:50 Two flavors of experimentation
09:45 Reading forty metrics at Uber Eats
13:10 Escaping the B2B feature factory
16:45 Anchoring the North Star to real usage
19:15 Where AI fits in research and analysis

Takeaways
- A losing experiment is often inconclusive, not negative; treat it as a map of the funnel rather than a verdict, and know when a big problem is worth another round.
- Persistence paid off at Codecademy: four months and three to four rounds of trial-model testing produced a 35% conversion increase.
- Separate your two experimentation modes; high-volume CRO chases many small wins, while big, uncertain bets are worth taking multiple shots to de-risk.
- Most B2B product teams are feature factories; the fix is a top-down OKR system, and planning usually breaks in the connections between layers, not inside them.
- Anchor retention and engagement to the product's natural use case, and use AI to synthesize research and simple A/B analysis in hours instead of weeks.

Connect with the Guest
Dan Layfield LinkedIn: https://www.linkedin.com/in/layfield/
About the guest: https://www.growthbook.io/podcast/guests/daniel-layfield
Episode page: https://www.growthbook.io/podcast/episode/1-25
Company Website: https://www.diligent.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:45 Meet Danielle Olean and Box's reinvention
02:45 Owning the entire customer life cycle
04:45 Why experimentation matters even without a checkout
07:45 The feature that's used but hidden
11:45 Proving ROI with a scrappy manual test
12:45 Building a culture that shares wins and losses
16:45 The pyramid strategy for prioritizing tests
18:45 The simplification tightrope on the pricing page
24:45 When a test wins for the wrong reason
27:45 Where experimentation at Box goes next

Takeaways
- Experimentation isn't only for e-commerce. Any product with a funnel, even an AI chatbot, can be measured and improved through testing.
- Simplification has a limit. Removing too much can strip away the cues and context buyers actually need to decide.
- Share losses as openly as wins. Wins build credibility, and losses build the psychological safety a testing culture runs on.
- Prioritize like a pyramid. Fix the widest-impact experiences first, then optimize down into smaller cohorts.
- Surprising results are the point. A test can win for a reason you never hypothesized, like the "wine effect," and that's where the real learning lives.

Connect with the Guest
Danielle Olean LinkedIn: https://www.linkedin.com/in/dolean1/
About the guest: https://www.growthbook.io/podcast/guests/danielle-olean
Episode page: https://www.growthbook.io/podcast/episode/1-26
Company Website: https://www.box.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Welcome and introducing James Falzone
01:45 What Kargo does and how real time ad auctions work
04:45 Why experimentation is embedded in Kargo's culture
07:45 The three things every marketplace has to deliver
10:15 The experiment that failed: click optimization on third party demand
12:15 A bad result versus a bad experiment
13:45 Why different customer types need different signals
15:30 Putting "where did you fail?" on every retro
18:45 How experimentation evolves with AI
21:15 Better not bigger: the closing takeaway

Takeaways
- A bad result is not a bad experiment. If you're not failing, you're probably not trying anything new.
- The same metrics and signals don't apply to every customer type. Bad results often come from a lack of context, not bad tech.
- Metrics and signals you test against should always be business driven, not ported from the last thing that worked.
- Put failure on the agenda. A biweekly "where did you fail?" retro turns one person's dead end into the whole team's shortcut.
- AI's biggest unlock is access. More people can run experiments, but it has to be built on solid ML and infrastructure. Better, not bigger.

Connect with the Guest
James Falzone LinkedIn: https://www.linkedin.com/in/jamesafalzone/
About the guest: https://www.growthbook.io/podcast/guests/james-falzone
Episode page: https://www.growthbook.io/podcast/episode/1-27
Company Website: https://kargo.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Welcome and what Fin actually does
02:00 How Fin became Anthropic's first line of support
02:30 Why Fin sells resolutions not deflections
06:00 Owning the stack with custom models
10:40 Pedro's path from fuzzy logic to AI
12:55 Why A/B testing is the only gold standard for AI
15:50 Do no harm testing on every change
18:00 The latency experiment that shocked the team
27:30 When more context made Fin hallucinate
30:15 Win rates and the future of AI driven experimentation

Takeaways
- Faster is not always better. Fin increased latency artificially and positive feedback went up, likely because a small delay makes an AI feel like it is doing real work.
- You cannot unit test a non-deterministic AI. A/B testing at scale, millions of samples in days, is the only reliable way to know a change actually helped.
- Adding more conversation history made Fin more helpful and more prone to fake promises, until a targeted prompt fix removed the hallucinations and kept most of the gain.
- A losing experiment is often a winner with one broken part. Diagnose which element hurts the experience, fix only that, and rerun.
- Fin A/B tests everything, even one-character prompt changes and many bug fixes, and treats a 20 to 30 percent win rate as a healthy sign of a real experimentation program.

Connect with the Guest
Pedro Tabacof LinkedIn: https://www.linkedin.com/in/tabacof/
About the guest: https://www.growthbook.io/podcast/guests/pedro-tabacof
Episode page: https://www.growthbook.io/podcast/episode/1-28
Company Website: https://fin.ai

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Introduction
01:25 From agency mass production to in house deep dives
04:05 Why some products resist selling online
06:35 The drop offs every ecommerce shop shares
07:45 The 50% win rate test Cogniteer reused
09:25 Why alignment beats developer resources
12:45 Two teams, two goals, one broken checkout
17:05 Selling water dispensers without a product catalog
22:15 Designing every experiment to lose
26:15 Personalizing buyers and where AI takes experimentation

Takeaways
- Deep dives beat mass produced tests, because understanding one business's users uncovers bigger levers than reusing the same test across many clients.
- Many ecommerce drop offs are structural, since the basket and product page leak in roughly 80% of shops because it is ecommerce, not because of your product.
- Product to channel fit decides what sells online, so books and fashion judge well on a screen while perfume and washing machines need cues the interface cannot fully provide.
- The real bottleneck is alignment, not developer resources, so agree on the problem and its hierarchy before anyone builds a variation.
- Match the interface to how people actually buy, because new buyers need information, returning buyers want speed, and B2B buyers often want a solution and an offer instead of a product catalog.

Connect with the Guest
Fabian Hans LinkedIn: https://www.linkedin.com/in/fabianhans-cogniteer/
About the guest: https://www.growthbook.io/podcast/guests/fabian-hans
Episode page: https://www.growthbook.io/podcast/episode/1-29
Company Website: https://www.cogniteer.de/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Cold open and introduction
01:45 Inside Farfetch, the global marketplace for luxury fashion
08:00 From startup validation to an experimentation mindset
09:45 A center of excellence that enables instead of executes
12:45 Fabs, build versus buy, and dropping the external vendor
16:45 One feature toggle as the entry point for every experiment
20:45 Learning rate over win rate
23:15 The two year experiment that replaced the recommendation vendor
29:45 Onboarding new product managers into experimentation
33:15 AI, corporate knowledge, and what comes next for experimentation

Takeaways
- Manage by learning rate, not win rate. The only failed test is one that was badly designed, with wrong metrics or sampling biases. Every other test produces a learning.
- Route every experiment through a single entry point. Farfetch's feature toggling system connects segmentation, user systems, CMS, and messaging so every team tests in the same language.
- External JavaScript injection tools carry hidden costs: broken pages, inconsistent results, and rework to reclaim your own data for deep dives.
- Strategic bets deserve a longer clock than fail fast allows. Farfetch iterated on its Inspire engine for two years before it beat and replaced the market leader.
- A center of excellence should enable, not execute. Farfetch's central team shrank while experiment volume grew because its job is ceremonies, templates, and coaching.

Connect with the Guest
Luis Trindade LinkedIn: https://www.linkedin.com/in/ltrindade/
About the guest: https://www.growthbook.io/podcast/guests/luis-trindade
Episode page: https://www.growthbook.io/podcast/episode/1-30
Company Website: https://www.farfetch.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Cold open and welcome
00:40 Vijay's path from Comcast to financial services
03:16 Making the experimentation platform self serve
05:08 The login widget experiment and the two second fallback
08:59 Documenting learnings from every experiment
10:38 AI in experimentation and the human in the loop
12:39 Advice for new product managers
15:24 Hypothesis driven metrics
18:25 Real time personalization and agentic AI
20:11 Democratizing experimentation with responsibility

Takeaways
- Self serve experimentation lets a small central team support a huge testing volume, but it only works with continuous training and guardrail metrics attached.
- In a regulated industry, every customer must be accounted for. Even one to two percent of users missing an experience is unacceptable.
- A simple fallback, like a two second load rule, can save an ambitious experiment without sacrificing coverage or security.
- Metrics should be driven by the experiment's hypothesis, not chosen by leadership in a silo. Pair a primary KPI with secondary KPIs for return behavior.
- AI saves real time in experiment analysis, but a human in the loop must validate anything AI produces before it goes live.

Connect with the Guest
Vijay Lal LinkedIn: https://www.linkedin.com/in/vijay-lal/
About the guest: https://www.growthbook.io/podcast/guests/vijay-lal
Episode page: https://www.growthbook.io/podcast/episode/1-31
Company Website: https://www.usbank.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Cold open and welcome
01:40 Erika's path from quantitative psychology to experimentation
03:35 When testing gets watered down
06:50 Experimentation in Principal's marketing space
09:55 The looping metric that finds stuck users
12:45 Building synthetic digital audiences
15:15 The first synthetic audience A/B test wins
22:30 The PDF lesson and meeting customers on mobile
25:15 North star metrics and the right contact cadence
28:05 Where experimentation goes next with AI agents

Takeaways
- Synthetic digital audiences let teams rank 20 content options by predicted engagement before spending a single live impression, and Principal's first synthetic selection beat the control in a real A/B test.
- A looping metric built from web behavior data can find where customers get stuck without heat-mapping tools: watch how often users cycle back to the same page within tight time windows.
- Testing gets watered down when "let's try something" replaces a control group; a little pre-planning gets far more out of every experiment.
- A center of excellence that shares wins and losses turns tribal knowledge into shared knowledge and stops "we tried that years ago" from killing valuable retests.
- An experimentation mindset requires that people can't get punished for mistakes; give teams guardrails and a safe playground and they'll stop running the same test forever.

Connect with the Guest
Erika Dunn LinkedIn: https://www.linkedin.com/in/erikadunn/
About the guest: https://www.growthbook.io/podcast/guests/erika-dunn
Episode page: https://www.growthbook.io/podcast/episode/1-32
Company Website: https://www.principal.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 introduction and Arie's path into experimentation
02:05 building programs at Kohl's, Marriott, and Total Wine
03:15 healthcare retail and 1,100 dentist owned offices
04:30 the test and learn team and 100 tests a year
07:15 building a culture of shared wins and explained losses
09:50 what a losing navigation redesign revealed
12:30 testing your way into big redesigns
13:55 the case for a 25% win rate
15:50 haircuts, holdouts, and honest math on stacked wins
17:55 metrics beyond conversion and where AI fits next

Takeaways
- Aspen Dental runs experimentation as healthcare retail: with 1,100 dentist owned offices, the office, not just the website visitor, is the real unit of analysis.
- A mature program should target a true win rate around 25%; a 40% win rate usually signals a young program still picking off low-hanging fruit.
- Apply a haircut to stacked wins: lifts depreciate as customers acclimate, and two 5% wins never add up to 10%.
- Losing tests are valuable when they're designed to isolate why: test your way into big redesigns instead of shipping them whole.
- Experimentation culture grows from sharing wins and explaining losses: keep the statistical rigor in the back end and the communication simple.

Connect with the Guest
Arie Polycarpou LinkedIn: https://www.linkedin.com/in/ariepolycarpou/
About the guest: https://www.growthbook.io/podcast/guests/arie-polycarpou
Episode page: https://www.growthbook.io/podcast/episode/1-33
Company Website: https://www.aspendental.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Introduction
01:40 Meet Edd Saunders and JobLeads
03:12 How JobLeads uses AI for prototyping
04:05 Building experimentation operations and a knowledge base
05:35 Learning over winning and compounding growth
08:10 The pizza personalization experiment
12:50 Moving from solution space to problem space
17:05 Problem mapping on a 2x2 matrix
22:20 Velocity and democratizing experimentation
24:20 AI automation for the unsexy work

Takeaways
- A one click reorder feature that cut a pizza ordering flow from 22 inputs to 5 had zero impact on purchases, proving that removing friction can also remove the customer's sense of control.
- Exploration is part of the customer's delight; returning customers wanted to browse the menu even though they ordered the same thing every week.
- Moving new experimenters from solution space to problem space thinking raises win rates and produces learnings the whole organization can use.
- Problem mapping on a 2x2 matrix of evidence versus impact turns customer research into a prioritized experiment roadmap, and one validated problem can spring a whole tree of testable ideas.
- Scaling experimentation from 0.3 to 2.8 tests per month is less about education and more about habit change, shared learnings, and giving non specialists the tools to launch their own experiments.

Connect with the Guest
Edd Saunders LinkedIn: https://www.linkedin.com/in/eddsaunders/
About the guest: https://www.growthbook.io/podcast/guests/edd-saunders
Episode page: https://www.growthbook.io/podcast/episode/1-34
Company Website: https://www.jobleads.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Cold open and welcome
01:30 The Clover business model and its scale
03:45 Ben's role and the metrics that matter
07:45 Deciding what gets tested: uncertainty and downside
09:05 Why Clover can't test in production
12:45 Testing the Shake Shack checkout experience
19:35 Advice for PMs new to experimentation
23:45 Context over personalization
25:45 The future of experimentation and the human element

Takeaways
- At Clover's scale, testing happens before rollout: pilots and detailed go-to-market plans replace in-production A/B tests, because a merchant's work tool can never change overnight without warning.
- Uncertainty and downside set the testing depth. High-risk changes like payment authorization flows earn deep, ground-level experimentation, while table stakes features like Apple Pay earn a monitored rollout.
- Testing is the evidence that justifies rollout investment: if the data doesn't show a feature will succeed, the go-to-market dollars never get spent.
- A checkout funnel can carry the brand. At Shake Shack, Ben's team tested prep-time expectations, fixed wrong-location orders, and used loading screens to deliver hospitality digitally.
- Structure experiments for durable business value, not pass-fail verdicts: pair headline metrics with counter metrics and anchor on ground-level measurements like items per check that resist marketing noise.

Connect with the Guest
Ben Schein LinkedIn: https://www.linkedin.com/in/benschein/
About the guest: https://www.growthbook.io/podcast/guests/ben-schein
Episode page: https://www.growthbook.io/podcast/episode/1-35
Company Website: https://www.clover.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Intro
01:10 About Supercell and 300 million players
02:30 The central team and a decentralized culture
04:25 Fewer than 100 tests a quarter
05:40 Sharing learnings across independent game teams
07:35 Retention as the North Star
08:35 Onboarding experiments with gems and tutorials
12:45 Telling players about A/B tests
15:15 Hypotheses and proxy metrics
20:45 AI and the future of experiment analysis

Takeaways
- Supercell runs fewer than 100 A/B tests a quarter for 300 million monthly players, because the goal is to become more hypothesis driven while staying creative, not to maximize volume.
- In a decentralized company, a central experimentation team earns its impact by providing the platform, partnering on rigor, and making sure learnings travel across independent game teams.
- Announce experiments to users before they run; Supercell's community managers tell players what is being tested and why, which turns a skeptical community into a research partner.
- Promise fairness, not just transparency; players who get the worse variant always receive a make-up event later, because game players come to have fun, not to be disadvantaged.
- AI-powered self-serve analysis means everyone can now run and analyze experiments, so the next challenge is making the quality of AI analysis consistent across the whole company.

Connect with the Guest
Shan Huang LinkedIn: https://www.linkedin.com/in/cnshanhuang/
About the guest: https://www.growthbook.io/podcast/guests/shan-huang
Episode page: https://www.growthbook.io/podcast/episode/1-36
Company Website: https://supercell.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:45 Meet Early Warning: fraud detection, Zelle, and a trillion dollars in payments
02:00 Wayfair and optimizing every step of the storefront funnel
03:05 The four questions to ask before any A/B test
05:15 Test setup best practices: hypotheses, guardrails, power, and pre-registration
07:40 Why 85 to 90% of tests fail and why that's a learning agenda
09:25 Novelty effects, hidden heterogeneity, and the multiple comparisons problem
12:55 Pre-registration, kill criteria, and stopping p-hacking
15:10 Rolling out winners: gradual ramps and long-running holdouts
17:15 Causal inference when you can't A/B test
20:05 The case for more A/B testing, not less

Takeaways
- Run the four-question framework before any test: clean randomization, a plausible effect size for your traffic, a reversible and cheap change, and a falsifiable hypothesis.
- Treat A/B testing as a learning agenda: 85 to 90% of tests are supposed to fail, and a suspiciously high win rate is a red flag, not a trophy.
- Pre-register the full analysis plan, including hypothesis, mechanism, primary metric, exact statistical test, and subgroups, so p-hacking can't creep in when a test goes sideways.
- Define kill criteria and success, failure, and guardrail-dip actions before launch, with leadership sign-off, so nobody chases a loss into a fake win.
- Log every test and its learnings where the whole organization can see them; that reinforcement loop is what separates world-class experimentation programs.

Connect with the Guest
Priya Singhee LinkedIn: https://www.linkedin.com/in/priya-singhee/
About the guest: https://www.growthbook.io/podcast/guests/priya-singhee
Episode page: https://www.growthbook.io/podcast/episode/1-37
Company Website: https://www.earlywarning.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Cold open and introduction
01:50 Shop It To Me and the first engineering hire
02:55 300 A/B tests as the path to one million subscribers
04:00 Testing as a core value and the second merge request
07:20 Learneo, Course Hero, and two ways to get access
08:50 Modeling lifetime value to stop teams competing
11:10 Testing the opposite of the hypothesis
14:35 A portfolio of small, medium, and large tests
17:15 Multi armed bandits and seasonal traffic
20:35 The flywheel that keeps copycats behind

Takeaways
- Test the opposite of every hypothesis. At Shop It To Me the inverse won surprisingly often, and even when it lost it proved the variable mattered.
- Keep tests small and focused. Redesign a whole page and lose, and you learn that version failed but not what to change next.
- Translate a growth goal into an execution count. One million subscribers is not actionable. 300 A/B tests by year end is, and everyone can influence it.
- Make experimentation part of hiring and onboarding. Every new engineer's second merge request was their own test idea.
- Analysis friction sets the ceiling on testing velocity. Three days of ad hoc SQL per test quietly discourages teams from running more.

Connect with the Guest
Rich Liebling LinkedIn: https://www.linkedin.com/in/richliebling/
About the guest: https://www.growthbook.io/podcast/guests/rich-liebling
Episode page: https://www.growthbook.io/podcast/episode/1-38
Company Website: https://www.learneo.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:45 About Zalando and its marketplace model
02:00 Mi's path from engineering to experimentation
03:10 Economists and data scientists in one decision-making org
04:15 Running over 1,000 experiments a year
05:55 What makes an experiment high risk
07:10 Sharing learnings through standardization and champions
09:10 The discovery feeds launch and its measurement plan
11:15 Balancing the experimentation portfolio
12:35 Growing a KPI tree from the North Star
16:05 LLM agents and the future of experimentation at Zalando
18:35 New missions for a longstanding business

Takeaways
- Treat experimentation as a portfolio: balance confirmatory tests that protect the business with game-changing bets that can win big.
- Assess risk tiers when building the roadmap so measurement rigor scales with the stakes instead of slowing every decision down.
- When a launch is too new to have a success metric, pair short term A/B tests with long term holdouts from day one.
- Connect every experiment to the company North Star by cascading it down to sensitive proxy metrics and controllable inputs.
- Use LLM based agents as a cheap simulation layer to screen hypotheses, not as a replacement for real A/B tests.

Connect with the Guest
Mi Tian LinkedIn: https://www.linkedin.com/in/mi-tian-941b4767/
About the guest: https://www.growthbook.io/podcast/guests/mi-tian
Episode page: https://www.growthbook.io/podcast/episode/1-39
Company Website: https://www.zalando.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Cold open and welcome
01:10 From growth hacker to Realtor.com
02:50 Three foundations for a new experimentation team
04:20 The 30/30/30 rule
05:05 The 300% bundling win that lost revenue
07:10 You don't need a stats degree to experiment
09:20 Cascading North Star metrics
12:50 Do the homework before the experiment
13:50 The wishlist: instrumentation, embedded knowledge, culture
15:50 AI as an aspirational data scientist
18:45 Keeping a human in the loop

Takeaways
- A winning decision metric is not enough. Realtor.com's bundling test hit a 300% attach rate, but funnel fallout from the extra step made it a net revenue loser. Set secondary metrics and their thresholds before launch.
- Expect the 30/30/30 rule: roughly a third of tests win, a third are inconclusive, and a third lose. The math is the math, and the losers carry most of the learning.
- Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments.
- You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing.
- AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test.

Connect with the Guest
Whitney Perez LinkedIn: https://www.linkedin.com/in/whitneykperez/
About the guest: https://www.growthbook.io/podcast/guests/whitney-perez
Episode page: https://www.growthbook.io/podcast/episode/1-40
Company Website: https://www.realtor.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:00 Intro
01:00 About Charlie Health
02:05 Joe's role across performance, lifecycle, and experimentation
03:30 Building experimentation rigor across teams
04:40 Even a loss is a win
05:15 The form page test that created a new bottleneck
08:50 Teachers Pay Teachers and the Easel lesson
16:30 Designing experiments that teach you something when they lose
21:00 Premortems for high-risk experiments
24:30 AI is flattening the org

Takeaways
- A winning test is a data point, not a finish line; Charlie Health's form page removal won on top-of-funnel metrics and still exposed a downstream bottleneck that became the next experiment.
- More traffic entering the funnel means conversion rate goes down, even when more people reach the bottom; treat it as a law of physics and plan for it before you read the results.
- Build it for everyone and you build it for nobody; Teachers Pay Teachers cut roughly 80% of the market for Easel, repositioned it around one job, and roughly quadrupled retention.
- Run a premortem before any large, risky experiment: ask the room "it was a massive failure, what went wrong?", have everyone write and share, and build the safeguards while there is still time.
- Before launching, check the quantitative and qualitative data behind the hypothesis, map the full funnel, and measure toward the real business outcome, so a loss still leaves you with a next step.

Connect with the Guest
Joe Yevoli LinkedIn: https://www.linkedin.com/in/joeyevoli/
About the guest: https://www.growthbook.io/podcast/guests/joe-yevoli
Episode page: https://www.growthbook.io/podcast/episode/1-41
Company Website: https://www.charliehealth.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Chapters
00:45 Meet Anuradha Tempe: from chip design to leading Samsung e-commerce
02:15 Why B2B is not a sidekick to B2C
03:30 Not everything needs an A/B test
05:15 Spreading experimentation practice across a global conglomerate
07:00 Bringing B2C rigor to a fast-paced B2B team
08:45 The add-on experiment that nearly doubled attach sales
11:15 Buyers decide on mobile and purchase on desktop
16:30 The Buy Now test that lost
21:30 North Star metrics, guardrails, and the EPP discount fix
26:05 An AI copilot for A/B test planning

Takeaways
- Treat B2B as its own customer base with its own testing discipline; a single bulk order on one day can inflate a B2B test into a false positive unless order data is normalized before results are read.
- Not everything needs an A/B test; route lower-risk changes through UAT feedback or pre/post comparisons and reserve full experiments for features where being wrong is expensive.
- Map where the decision happens, not just where the purchase happens; Samsung's business buyers decide on mobile and buy on desktop, and surfacing add-ons on mobile nearly doubled attach sales.
- B2B buyers value clarity over faster conversions; a Buy Now button earlier in the flow confused bulk purchasers because returns and cancellations on large orders are costly.
- Align on the North Star before building, whether it is revenue, engagement, NPS, or fewer support tickets, and set guardrails so an engagement feature can never quietly drag sales down.

Connect with the Guest
Anuradha Tempe LinkedIn: https://www.linkedin.com/in/anuradha-tempe/
About the guest: https://www.growthbook.io/podcast/guests/anuradha-tempe
Episode page: https://www.growthbook.io/podcast/episode/1-42
Company Website: https://www.samsung.com/us/business/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to http://growthbook.io

Will shares the Barnes & Noble order confirmation test he was sure would lose, and why the same idea never worked for any other client. He walks through a recent GoPro Mission launch test that asked whether a step-by-step configurator adds too much friction, and what a flat result revealed about high consideration buyers. He also explains why GoPro shares interim readouts across the company, why win rate makes a poor North Star for an experimentation program, and how his team plans to use AI for speed without outrunning its own learnings.

Chapters
00:00 Intro
00:52 Will's role running e-commerce at GoPro
01:43 Learning A/B testing across retail at Brooks Bell
05:24 How GoPro runs one to three tests a month
06:35 Sharing learnings and interim readouts across teams
09:20 The Barnes & Noble order confirmation win
13:02 Why the win did not transfer to other clients
14:41 Testing friction on the GoPro Mission configurator
18:51 Why win rate is the wrong North Star
22:19 How AI will shape experimentation at GoPro

Takeaways
- A winning idea rarely travels. The Barnes & Noble recommendation module worked because of that audience's low order values and reading habits, and it failed for every other client that tried it.
- Design every test so it teaches you something whether it wins, loses or ends flat. Losing tests are jet fuel when the learning is built in.
- A flat result is still an answer. GoPro's configurator test showed that buyers of high consideration products accept extra steps when each choice adds value.
- Share interim readouts across the company, and use them to show how volatile results are before a test reaches statistical significance.
- AI can speed up building and running experiments, but a team that runs more tests than it can learn from is not getting better.

Connect with the Guest
Will Guyeskey LinkedIn: https://www.linkedin.com/in/willguyeskey/
About the guest: https://www.growthbook.io/podcast/guests/will-guyeskey
Episode page: https://www.growthbook.io/podcast/episode/1-43
Company Website: https://gopro.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off decision framework comes from. She also explains the parts of B2B experimentation that have no B2C equivalent: choosing between user-level and account-level randomization, the interference risk when two colleagues at the same company see different pricing, and the many-to-many problem of one developer belonging to several accounts. And because Twilio's customer counts are lower than a consumer business, her teams often ship on signals and team conviction rather than waiting for statistical significance.

Chapters
00:00 Cold open
01:02 Welcome Wanli Lau of Twilio
01:23 Inside Wanli's role at Twilio
04:03 Eleven years and 1,500 tests a year at Expedia
04:58 Why B2B experimentation differs from B2C
05:33 Cross-functional collaboration and good hypotheses
06:39 Teaching teams to experiment rigorously
07:48 The Expedia sign-up takeover that won on conversion
11:09 Guardrail metrics and the trade-off framework
14:52 Designing experiments to maximize learning
17:46 Diagnosing where a feature failed
19:10 North Star metrics and account-level randomization at Twilio
21:40 Learning from signals instead of significance
22:11 Where experimentation goes next at Twilio

Takeaways
- A test that wins on the primary metric can still be a loss. Expedia's sign-up takeover raised conversion and damaged return rate and engagement at the same time.
- Name the primary, secondary and guardrail metrics before the test runs, not at readout. Deciding afterwards turns a result into a debate.
- Conversion should be a do-no-harm guardrail for teams that do not own it, even when it is not their primary metric.
- In B2B the randomization unit is a design decision. User-level bucketing risks two colleagues at one account seeing different prices, and account-level avoids that but costs sample size.
- When customer counts are low, significance is often out of reach. Learning from signals and shipping on team conviction beats waiting for a number that will never arrive.

Connect with the Guest
Wanli Lau LinkedIn: https://www.linkedin.com/in/wanlilau
About the guest: https://www.growthbook.io/podcast/guests/wanli-lau
Episode page: https://www.growthbook.io/podcast/episode/1-44
Company Website: https://www.twilio.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it.

Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late.

Chapters
00:00 What customers actually care about
01:14 Meet Ashraf Karim of ServiceNow
01:39 Owning the entire post-sale experience
02:08 Why ServiceNow runs as its own Customer Zero
03:02 Lessons from Google, PayPal and Verizon
07:11 The agentic skills users kept abandoning
09:29 Latency, expectations and the feedback loop fix
12:29 Users wanted the outcome, not the process
14:49 Designing experiments that can afford to lose
18:13 Using an LLM judge on nondeterministic models
21:48 Customer effort score as a guardrail metric
25:03 Where experimentation at ServiceNow goes next

Takeaways
- ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it.
- The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like.
- The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs.
- Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps.
- Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer.

Connect with the Guest
Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn
About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim
Episode page: https://www.growthbook.io/podcast/episode/1-45
Company Website: ServiceNow

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

Chapters
00:00 Cold open
00:54 Welcome Gaurav Sethi of PayPal
01:30 Elmo, PayPal's homegrown experimentation platform
03:53 $180 million in revenue impact and cost avoidance
04:28 The win rate that was too good to be true
06:29 Instrumentation standards and the exposure event
08:56 The carousel test: 700 days down to 51
11:29 Designing experiments so every result teaches you
13:45 Building the platform is only half the job
15:07 Education, office hours and executive support
18:28 Exposure events and joining transactional data
24:14 AI for experimentation, and experimentation for AI

Takeaways
- A win rate of 55% to 60% against an industry average of 11% to 14% was a tracking problem, not a performance one: assignment data carried 25% to 30% dilution and reached significance on users who never saw the test.
- The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load.
- Standardized instrumentation and canonical metric definitions are what let the analysis pipeline run without human cleanup, taking experiment readouts from four weeks to 24 hours.
- A six variant carousel test projected at 700 days was resolved by a multi armed bandit in 51 days, with a winner worth almost five basis points.
- About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments.

Connect with the Guest
Gaurav Sethi LinkedIn: https://www.linkedin.com/in/gauravsethi22
About the guest: https://www.growthbook.io/podcast/guests/gaurav-sethi
Episode page: https://www.growthbook.io/podcast/episode/1-46
Company Website: https://www.paypal.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

She walks Ashley Stirrup through the test that changed how her team thinks about discovery. MilliporeSigma rebuilt its on-site type ahead so it suggests a search term instead of pushing a matching product, and saw double digit increases in feature usage and add to cart. From there Dorothy makes the case for measuring the action a page is actually responsible for rather than revenue alone, for pairing A/B tests with a user testing ambassador panel to get at the why, and for sending product managers back to one question before anything gets built: what problem are you solving for the customer?

Chapters
00:00 Revenue is an outcome, not the metric
01:26 Welcome and what MilliporeSigma does
02:22 The three audiences on a B2B science site
03:53 What an A/B testing analyst coordinates
04:20 Running experiments across product pods
05:31 Building leadership buy in for testing
08:08 Where A/B testing ends and user testing begins
08:52 The type ahead search test that lifted add to cart
11:30 How to frame a test with a product manager
13:34 Why testing the opposite pays off
15:26 Personas, personalization and what AI changes
18:57 Building trust across competing teams

Takeaways
- MilliporeSigma changed its on site search to suggest search terms instead of products and saw double digit gains in feature usage and add to cart.
- Revenue is an outcome, so Dorothy's team measures the next action each page is responsible for, like add to cart on a product detail page.
- A/B testing tells you what happened. MilliporeSigma's user testing ambassador panel supplies the why behind the click.
- Every test starts with one question to the product manager: what problem are you solving for the customer, not what is your hypothesis.
- Dorothy is building personas into agents so browsers, not just buyers, can be tested for a directional read before an experiment ships.

Connect with the Guest
Dorothy Crepin LinkedIn: https://www.linkedin.com/in/dorothycrepin/
About the guest: https://www.growthbook.io/podcast/guests/dorothy-crepin
Episode page: https://www.growthbook.io/podcast/episode/1-47
Company Website: https://www.sigmaaldrich.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes, and a concept test in the merchant promotions space that sent the team toward a much simpler dashboard than their research predicted. Michal also explains why she follows losing experiments with user interviews, why "I didn't even notice it" is valuable feedback, and why every company should decide deliberately whether it de-risks upstream or in a fast follow after launch.

Chapters
00:00 Where design is heading in the AI era
00:53 Meet Michal Lenik of Grubhub
02:21 Design as an upstream business lever
03:48 Small cohorts instead of classic A/B tests
05:27 The merchant concept test that changed direction
08:18 Prototyping with AI and merchant round tables
10:16 Success metrics for product, design and engineering
11:56 Pairing experiment data with user interviews
14:26 AI, personalization and the designer's new role
17:45 De-risking upstream versus shipping fast

Takeaways
- Grubhub tests big bets by releasing to internal users or small cohorts first, so it can read business impact without disrupting core revenue and order metrics.
- A concept test with merchants overturned what fifteen research interviews suggested, steering Grubhub to a simpler dashboard and away from a launch that likely would not have won.
- AI lets Grubhub's design teams put clickable prototypes in front of users early, turning each iteration into a micro experiment.
- Pairing quantitative results with five to ten user interviews after a test explains why it underperformed, including whether users even saw the change.
- Both upstream de-risking and fast iteration work. The right choice depends on how quickly engineering can ship, and it should be made on purpose.

Connect with the Guest
Michal Lenik LinkedIn: https://www.linkedin.com/in/michallenik/
About the guest: https://www.growthbook.io/podcast/guests/michal-lenik
Episode page: https://www.growthbook.io/podcast/episode/1-48
Company Website: https://www.grubhub.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

• (00:00) - Where design is heading in the AI era
• (00:53) - Meet Michal Lenik of Grubhub
• (02:21) - Design as an upstream business lever
• (03:48) - Small cohorts instead of classic A/B tests
• (05:27) - The merchant concept test that changed direction
• (08:18) - Prototyping with AI and merchant round tables
• (10:16) - Success metrics for product, design and engineering
• (11:56) - Pairing experiment data with user interviews
• (14:26) - AI, personalization and the designer's new role
• (17:45) - De-risking upstream versus shipping fast

The conversation covers the churn run that lifted F1 by 6% and is worth about a million dollars over baseline, the two popular parlays tests that became penalty weights and floors in the next experiment, North Star versus OEC in a luck-contaminated business, responsible gaming as a guardrail a machine has to be taught, and Michael's six-level ladder of experiment autonomy, including the shadow mode rung where FanDuel buys trust with six months of evidence. For product, data and growth leaders deciding how much of an experiment a machine should be allowed to run.

Chapters
00:00 Cold open
01:02 FanDuel, prediction markets and the data and AI product team
03:23 The auto researcher: 300 strategies, an hour, $300 in tokens
05:25 Three surprises: it builds features, enforces the holdout, reframes the problem
07:50 Two popular parlays tests and the penalty weight they became
12:38 What humans still own: the hypothesis and the goal
16:00 North Star vs OEC and luck-adjusted margin
20:18 The churn run: 750 features to 20 and a million dollars over baseline
24:18 The six levels of experiment autonomy and shadow mode
36:32 Always measure baseline, and when to stop iterating

Takeaways
- FanDuel's auto researcher reads an objective function, proposes a strategy, writes the model, scores it and repeats about 300 times in an hour for around $300, returning the winner as readable code.
- The agent builds its own features (756 candidates to 20), never sees the validation set, and will reframe a problem when the scoring rules make a wrong guess cost more than a right one earns.
- One churn run cut 750 features to 20, lifted F1 by 6% and tested live across three arms of 47,000 users, worth roughly a million dollars over baseline.
- Losing tests become coded constraints: two losing popular parlays tests turned into a 2.5x penalty weight and a hard floor in the next experiment's envelope.
- Autonomy is earned in shadow mode, where the machine's complete package is recorded against every human decision until six months of evidence settles the trust question.

Connect with the Guest
Michael Mathews LinkedIn: https://www.linkedin.com/in/michael-mathews-jr-0b194224
Company Website: https://www.fanduel.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Host Ashley Stirrup also hears the story of a localized payment method the whole company expected to win, why it lost to PayPal and cards, and how the team used Stripe data and other sources of truth to prove it. Sapna closes with how VidaXL designs experiments, from guardrail metrics and device segmentation to the 70% of the work that happens before launch. For CRO, product and growth teams building an experimentation program that the business trusts.

Chapters
00:00 Intro
01:35 VidaXL: 30 countries, every team tests
02:28 Sapna's role and the maturity program
04:39 The 38.7% B2B signup winner
08:42 Designing experiments: research, guardrails, device segments
12:07 The localized payment method that lost
15:58 Seven years of maturing
16:53 70% of the work is before launch
18:33 Wrap

Takeaways
- Friction is not always bad: buyers dropped off a registration page because they could not see the subscription details, and adding them lifted account creation 38.7%
- Localization is not a guaranteed win; a country-specific payment method lost to PayPal and cards, and only cross-verified data got the team to accept it
- Guardrail metrics (return rate, latency) are decided with the stakeholder before launch, not after
- Segment by device and traffic source; a change that works on desktop is not shipped on mobile if it did not win there
- About 70% of an experiment's work is done before it goes live: research, clear objectives, data quality, then the analysis is the small part

Connect with the Guest
Sapna leads and manages roadmaps for 155+ growth initiatives and experiments a year, has helped build and scale an experimentation culture across the organization, and has driven CRO and experimentation across both B2B and ecommerce businesses.
Sapna Kashyap LinkedIn: https://www.linkedin.com/in/sapnakashyapx/
Sapna Kashyap Website: https://sapnakashyap.com/

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Host Ashley Stirrup also hears the story of a cross-sell test everyone expected to win: showing complementary items on searches like banana encouraged exploration, but that engagement did not consistently translate into the conversion outcomes the team had hoped for. That highlighted an opportunity to better align the experience with shopper intent, so the team rebuilt around the customer's mission. The rebuilt version delivered better CTR and better conversion, and the team submitted a white paper on the work to the AAAI 2027 Innovative Applications of AI conference for consideration. Arv closes with how to design experiments for learnings instead of launches and why self-serve experimentation is the next step beyond the analyst. For product, analytics and retail media teams who run experiments on shared pages.

Chapters
00:00 Intro
01:42 Balancing metrics across search, shopping agents and advertisers
03:08 Guardrails, degradation thresholds and long-term holdouts
05:58 The banana problem: rethinking a cross-sell idea
08:20 Rebuilding around the customer's mission
11:16 Designing experiments for learnings, not launches
13:48 Trustworthy experiments and where experimentation goes next
15:56 Wrap

Takeaways
- Every page owner has their own primary metric; ads teams negotiate a degradation threshold per page and stay inside it
- Revenue is not the only guardrail: relevance and latency are long-term metrics, and ad-free holdouts running for months measure what ads cost in trust
- Customers engaged with complementary recommendations, but that engagement did not always translate into purchases; the fix was reducing diversity and building themes around the customer's mission
- Start every experiment by asking how to make the product more usable, not how to launch it; the hypothesis changes and so do the metrics
- A written process (test plan, no early peeking, end-of-test checks) catches bugs early and builds trust; self-serve tooling is what lets PMs experiment without an analyst

Connect with the Guest
Arv Ramchandani LinkedIn: https://www.linkedin.com/in/arv-ramchandani
Company Website: https://www.walmartconnect.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

Mary also shares two hard-earned lessons from earlier in her career: an 18 month redesign that taught her to test early and in small pieces, and a payment processor test where the randomization was not what it seemed. She closes with how Headspace is scaling experimentation and why every portfolio needs a learning bucket. For data, product and growth leaders experimenting on AI features and long term outcomes.

Chapters
00:00 Cold open
01:02 Headspace and Mary's path from Amazon and HBO
03:26 Why more engagement is not always better
09:25 Predicting long term outcomes with composite metrics
13:29 Testing Ebb: inputs, outputs and guardrails
19:28 A/B testing LLM features
22:35 Office hours and a culture of experimentation
28:46 The redesign that waited too long to test
32:03 Know your treatment: the payment processor test
38:11 Faster tests and the learning bucket

Takeaways
- At Headspace the target is health outcomes, not time in the app, so engagement metrics sit alongside self-reported measures of how members feel
- Every experiment on Ebb, Headspace's AI companion, is defined by inputs, outputs and guardrails, with clinicians doing clinical QA
- Stop rules written before launch let teams self-serve: lift with no guardrail harm ships, guardrail harm rolls back
- Test early and in discrete pieces; an 18 month redesign could not show which change was underperforming
- Keep a learning bucket of experiments that never have to report a win, alongside small loop tests and big bets

Connect with the Guest
Company Website: https://www.headspace.com

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io