In this episode of IntelliJAMS, Alex and Adrian dig into the "Frequently Unasked Questions" framework: four categories of questions shoppers silently ask themselves while browsing your site. They cover where most Shopify brands fall short on messaging, why reducing friction is easier than building motivation (but both matter), when urgency tactics actually work versus when they erode trust, and how to build hypotheses around messaging that you can test and learn from.

Timestamps:
0:00 - Intro and the Frequently Unasked Questions framework
0:44 - Why "unasked" questions matter more than FAQs
1:57 - The four categories: understanding, motivation, difference, trust
4:26 - Motivation vs. friction and how they drive behavior
6:35 - Why motivation is harder to build than friction is to reduce
7:40 - Urgency as a bonus category (and when it gets hacky)
9:56 - Fake scarcity vs. real scarcity: the windscreen wiper example
11:22 - Difference: competing within your range, against competitors, and against inertia
14:09 - How to figure out which unasked questions to prioritize
17:13 - Message, expression, and placement: the three layers of testing
19:02 - Why one test usually leads to five more questions
20:15 - Where to start: trust is the fastest win, difference is the biggest opportunity
24:07 - Recap and where to find Adrian

Topics covered:

The Frequently Unasked Questions framework (understanding, motivation, difference, trust)

Why "difference" is the most overlooked messaging gap on product pages

The motivation-to-friction ratio and how it affects conversion

When urgency and scarcity tactics help vs. hurt your brand

How traffic source (paid social vs. search) should shape your messaging strategy

Building messaging hypotheses you can A/B test

Message vs. expression vs. placement as three layers of experimentation

Why trust is the fastest win for most e-commerce brands

Ready to start testing your messaging? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535

Connect with Adrian / Scale Messaging:

Website: https://scalemessaging.com
LinkedIn: https://www.linkedin.com/in/adrianjstewart/

Connect with Intelligems:

Website: https://intelligems.io
Blog: https://intelligems.io/blog
LinkedIn: https://www.linkedin.com/company/intelligems

In this episode, we break down why post-purchase might be the highest-leverage Shopify A/B testing opportunity most brands haven't explored yet.
In this episode of IntelliJAMS, Adam and Alex explore the economics of post-purchase upsells — from a CBD brand that grew AOV 20% overnight to the consumption psychology that makes "buy more now" actually better for long-term LTV.

Timestamps:
0:00 - Intro
0:22 - Why post-purchase upsells are worth your attention
1:04 - "Be close to the money" — does post-purchase qualify?
1:52 - What brands are doing with post-purchase today
2:11 - The CBD gummies case study: 20% AOV lift overnight
3:32 - Why there's zero downward pressure on conversion
5:05 - The consumption psychology surprise: more supply = faster consumption
6:09 - Scarcity vs. abundance mindset and second-order effects
8:04 - Three post-purchase upsell strategies that work
9:35 - Matching upsell strategy to your margin profile
10:26 - Using post-purchase for inventory clearance and sell-through
11:02 - How Intelligems handles post-purchase testing and measurement
12:28 - Why post-purchase tests can run in parallel with everything else

Topics covered:

The economics behind post-purchase upsells (zero incremental CAC, no conversion risk)

A real-world CBD brand case study: 50% off second pack, 40% take rate, 20% AOV increase

Three post-purchase strategies: same product discount, complementary products, and clearance/inventory sell-through

Why consumption scales with supply — and what that means for repurchase rates

How to match your upsell strategy to your margin profile

Running post-purchase tests in parallel without interaction effects

How Intelligems measures incrementality, revenue, and profit on post-purchase offers

Want to start testing post-purchase upsells (or anything else)? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535

Connect with Intelligems:

Website: https://intelligems.io
Blog: https://intelligems.io/blog
LinkedIn: https://www.linkedin.com/company/intelligems

In this episode of IntelliJAMS, Adam and Alex explore the difference between test output and test outcomes, how to run concurrent tests without confounding your results, and why elevating the conversation from "number of tests" to "strategic priorities" changes everything.

Timestamps:
0:00 - Intro
0:27 - The question everyone's asking right now
0:52 - Why "how many tests" is the wrong question
2:23 - Outputs vs. outcomes: reframing productivity
3:10 - Test the plan, don't plan the tests
3:51 - You can't predict how many tests it'll take
4:54 - Running concurrent tests without confounding results
5:53 - Scoping metrics to the right part of the funnel
6:48 - Every test needs a hypothesis
7:30 - One test often leads to five new questions
8:15 - Real example: ending a checkout upsell test early
9:11 - How to push back on "I need 10 tests this month"
11:49 - Elevating the conversation from output to strategy
13:28 - Final thoughts: see the forest for the trees

Topics covered:

Why counting tests is an output metric, not an outcomes metric

How to reframe testing around strategic business goals

Running concurrent A/B tests on Shopify without confounding data

Segmenting tests by funnel stage and visitor type

How one test can lead to five new questions

Pushing back when stakeholders demand a test quota

Real-world example: killing a checkout upsell test 24 hours in

The "test the plan, don't plan the tests" framework

Want to sharpen your experimentation skills? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535

Website: https://intelligems.io
Blog: https://intelligems.io/blog
LinkedIn: https://www.linkedin.com/company/intelligems

In this episode of IntelliJAMS, Alex McEachern and experimentation expert Ally Petretti walk through the most common anxieties of A/B testing and how to manage them with better process instead of more stress.

Should you check your A/B test results every day?
Checking your test daily is like weighing yourself multiple times a day on a diet. The number isn't wrong, but it's not ready to be meaningful yet. Check to make sure the test is collecting data and firing correctly, but don't read into the directional results until you've hit your planned runtime and traffic thresholds.

What is statistical significance and when should you end an A/B test?
Statistical significance is often treated as a finish line, but reaching stat sig on day one doesn't mean you have a winner. Stat sig only looks at the math. You also need time (to account for day-of-week behavior changes, promotions, and the novelty effect) and volume (small sample sizes are fragile and can skew results dramatically). A test typically needs to run for at least one to two full weeks to account for these variables.

What is the novelty effect in A/B testing?
The novelty effect happens when returning visitors see something different on your site and react to the change itself rather than the actual experience. In price testing, this can look like a customer treating a different price as a discount and converting out of urgency. The novelty effect fades after a week or two as your sample includes more new visitors and repeat visitors adjust.

What is metric shopping in CRO?
Metric shopping is when you pick a winning metric after a test is already running, instead of sticking to the success metric defined in your hypothesis. For example, if your hypothesis targeted conversion rate but AOV happened to spike, declaring a win based on AOV is metric shopping. It's what Ally calls "the silent killer of CRO." The right move is to note the unexpected metric change, form a new hypothesis around it, and run a separate test.

How should you communicate A/B test results?
Don't just drop a list of metrics. That's left open to interpretation. Data storytelling means framing results around your original hypothesis, explaining what customers did and why, and tailoring the narrative for your audience. A CFO needs a different story than a CMO. Use tools like the Intelligems Slack bot or MCP integration to help frame results for different stakeholders.

How do you handle test ideas that aren't backed by data?
Ideas from teammates or Twitter threads aren't bad, but they need a hypothesis before they become tests. A strong hypothesis includes a data-backed problem, a proposed solution (the test idea), and a success metric. If an idea doesn't have that, either formulate the hypothesis yourself from existing data or add it to the backlog until you can.

How do you coordinate A/B testing across teams?
CRO teams and paid media teams often test independently, which creates noise in each other's data. The key is sharing hypotheses (not just test plans), coordinating testing calendars, and recognizing that every team is working toward the same growth goal. Regular cross-team syncs where you discuss challenges, upcoming campaigns, and testing angles can turn conflict into collaboration.

Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-aca...

Connect with Ally Petretti:
https://www.linkedin.com/in/ally-petretti-kuhn-47a27014/ 

Connect with Intelligems:

Website: https://intelligems.io
Blog: https://intelligems.io/blog

----
0:00 - Intro: Why we need a CRO therapist
1:05 - The #1 stressor in testing: urgency and timing
1:44 - "I'm a genius" vs. "I'm killing the business" (early test reactions)
2:33 - Peeking at tests: checking vs. reacting
4:51 - Pre-test analysis: setting guardrails before you launch
5:30 - Statistical significance isn't the finish line
6:13 - The novelty effect and why it matters for price testing
7:47 - Using the time series chart to read test stability
8:39 - When to call a test early (the right way)
10:12 - No test losers: every result is a learning
10:59 - Hypothesis discipline and metric shopping
12:18 - What metric shopping is and why it's the silent killer of CRO
13:25 - Communicating test results: data storytelling
15:57 - The three E's framework: Explore, Experiment, Extend
17:25 - Understanding your customers through testing
18:20 - "Feelings over data": handling test ideas without a hypothesis
21:19 - Cross-team testing: getting paid media and CRO on the same page
25:42 - Wrap-up and where to connect with Ally

In this episode, Alex and Amanda explore what happens when you balance conversion rate, AOV, and subscription adoption in every test instead of picking one, why the old UX tricks (spin-to-wins, slashed prices, CTA color swaps) are losing their edge, and how to get buy-in for short-term metric dips that lead to long-term growth.

Timestamps:
0:00 - Intro and why Amanda calls herself "a little crazy"
1:20 - The limits of traditional CRO and flat testing
3:00 - What's changed in the last 10 years of e-commerce UX
4:17 - Consumers are catching on to optimization tricks
6:10 - Why reducing discounts is actually improving conversion
7:36 - Shifting from product sales to business strategy (bundles, subscriptions)
9:02 - No silver bullets: what works for your business vs. someone else's
10:20 - Conversion rate is one-dimensional and easy to game
11:43 - The growth experiment mindset: testing for business growth
13:58 - Intentional trade-offs and why short-term losses can be worth it
15:10 - The two most important questions in commerce
16:19 - Getting buy-in when a metric temporarily looks bad
19:15 - "Testing is a tool, not a job title"
21:14 - Diminishing returns of small, isolated tests
22:58 - "Don't plan your tests. Test your plan."
24:03 - Amanda's advice: be bold, back-plan from ideal outcomes
25:39 - Why testing has to be cyclical, not one-and-done
27:06 - Where to connect with Amanda

Topics covered:

Why conversion rate optimization (CRO) is too narrow for modern e-commerce

Balancing conversion rate, AOV, and subscription adoption in a single test
How consumer behavior has evolved and why old UX tricks are losing effectiveness

The counterintuitive finding that reducing discounts can improve conversion
Using A/B testing and experimentation to validate business strategy, not just website changes

Getting executive buy-in for short-term metric trade-offs

The growth experiment mindset for Shopify brands

Why testing needs to be cyclical and revisited over time

---

Want to think bigger about your testing program? Join GEM Academy for free courses and a community of e-commerce operators sharing what actually works: https://www.skool.com/intelligems-academy-1535

Connect with Amanda Siegel: https://www.linkedin.com/in/amandadsiegel/

Connect with Intelligems:

Website: https://intelligems.io
Blog: https://intelligems.io/blog
LinkedIn: https://www.linkedin.com/company/intelligems