All insights
EN · CRM, Lifecycle & Personalization

Your CRM A/B test found a winner. Did the journey create incremental conversion?

A/B selects the better experience among treated customers. An eligible no-journey holdout answers whether the automation added an outcome.

Three parallel paths carry customer tokens to outcome trays; two pass through message gates while the third bypasses the journey.
A and B select the better experience; the no-journey path shows whether the intervention added an outcome. · Generated with OpenAI

Direct answer: assign three groups before the first message. A and B receive different versions of the same journey; an eligible holdout does not enter that journey during the defined window. A versus B selects a treatment. A or B versus holdout estimates the journey's effect. If a group called control receives the default version or merely waits to receive the winner, it does not answer the second question.

Platform vocabulary can hide the distinction. Customer Insights calls version A a control and documents an initial test audience that can receive the winner later. Salesforce randomizes journey paths. Adobe's documented journey holdout excludes profiles from all journey communication and compares outcomes, while also noting availability, sample, and cross-journey limits. Put the design in the experiment contract, not only in a platform setting.

Use a three-way journey proof

  • Frozen eligibility — who could enter, when, under which product, market, channel, and exclusion rules.
  • Stable assignment — person or account ID, experiment version, A, B, or holdout, and a timestamp before exposure.
  • Treatment A — the current journey, including message, timing, channel, offer, cost, and policy.
  • Treatment B — one bounded change; do not change content, channel, audience, and incentive together.
  • No-journey holdout — service and transactional messages continue, but the tested intervention does not.
  • Actual exposure — sends, deliveries, failures, other journeys, media, and sales contact.
  • Mature outcome — purchase, opportunity, renewal, revenue, or contribution with a window, corrections, and eligible denominator.
  • Decision — promote A or B, keep the current rule, redesign, or stop, with an owner and date.

Assign at person level when treatment and contamination are individual. For ABM, franchises, or shared purchases, use account, store, or territory when people in the same unit must not receive conflicting treatments. Randomization does not guarantee equal counts; NIST and platform documentation treat realized allocation as probabilistic. Stratify only on variables chosen before the test and report imbalance.

Separate four questions that dashboards blend

  • Execution — did the journey enter, deliver, and exit as designed?
  • Optimization — did B outperform A on the selected outcome?
  • Incrementality — did treatment outperform eligible people who received no journey?
  • Economics — did incremental contribution cover messaging, incentive, media, data, review, and operations?

Clicks and opens help debug delivery, but a no-message holdout cannot compete on opens. Use a downstream outcome compatible with the buying cycle. A native journey goal is operationally useful, yet it may apply its own counting and attribution window. Reconcile the event to CRM, commerce, and, where the decision requires it, the revenue rung in https://makinai.co/insights/en/crm-closed-won-booked-billed-collected-revenue-ai.

Illustrative U.S. example

A hypothetical midsize subscription company has 30,000 customers eligible for a second-purchase journey. It assigns 12,000 to the current rule, 12,000 to an AI-assisted recommendation, and 6,000 to no journey. After the purchase and return window, A converts at 5.0%, B at 5.4%, and holdout at 4.8%. B beat A by 0.4 percentage point, but its difference versus holdout is 0.6 point; A added 0.2 point. Only then should the team apply contribution, discounts, contact cost, and uncertainty. This is an illustration, not a client case or benchmark.

If holdout received a parallel promotion or some customers in B never received the message, analyze by original assignment and show actual exposure separately. Removing undelivered records after randomization favors treatment. The primary analysis should keep all eligible units in their assigned arm; exposure-based views are diagnostic.

Implement in six weeks

  • Week 1 — select one frequent, reversible journey; define unit, eligibility, intervention, outcome, window, and smallest commercially useful effect.
  • Week 2 — map other journeys, sales, media, consent, transactional messages, exclusions, and the authoritative outcome source.
  • Week 3 — implement persistent assignment plus entry, exposure, exclusion, outcome, and correction events; test duplication and reentry.
  • Week 4 — run shadow assignment while the current journey remains unchanged; reconcile counts and identities.
  • Week 5 — release A, B, and holdout at approved volume; monitor safety, contact pressure, delivery failure, and contamination.
  • Week 6 — read only the first available window; do not declare a winner before revenue or returns mature; lock the final read date.

Size investment and trade-offs

Cost rises with journey and channel count, cross-device identity, assignment unit, reentry, duration, late outcomes, offline purchase, returns, consent, external data, campaign overlap, minimum volume, and the need for a global holdout. A journey-level holdout is simpler and answers one intervention. A persistent cross-journey holdout estimates the portfolio but requires shared orchestration and withholds more treatment.

Do not promise a universal holdout percentage. Size and duration depend on baseline conversion, variance, minimum useful effect, unit, clustering, outcome maturity, and the cost of a wrong decision. Small populations may support only A versus the current rule, a sequence of tests, or a quasi-experiment. An inconclusive result is evidence; it is not permission to select the prettier version.

Define rules, AI, and accountability

Rules own eligibility, consent, assignment, exclusions, transactional classes, frequency, windows, and stopping. AI can develop variants, estimate heterogeneity after the test, detect contamination, and propose hypotheses. It should not change arms, outcome, or window mid-test, remove unfavorable cases, or promote a click winner when the decision is revenue. Marketing owns the hypothesis; CRM/MarTech, delivery; Data, randomization and analysis; Sales and Service, parallel contact; Finance, mature value; leadership, holdout cost and release.

Release value, not just a winner

  • Eligible coverage, stable assignment, baseline balance, and improper reentry.
  • Delivery and contamination by arm, channel, journey, and sales contact.
  • B minus A and treated minus holdout, with uncertainty and absolute effect size.
  • Incremental revenue, contribution, and cost per eligible unit after returns and maturity.
  • Opt-outs, complaints, contact pressure, service cases, and approved segment effects.
  • Time to decision, experiment cost, and decisions changed by the evidence.

Coordinate the portfolio through https://makinai.co/insights/en/ai-marketing-capacity-valid-experiment-operating-system, reconcile cross-channel exposure with https://makinai.co/insights/en/ai-cross-channel-contact-pressure-crm-frequency, and mature the financial outcome through https://makinai.co/insights/en/crm-closed-won-booked-billed-collected-revenue-ai. If A/B tests select messages but the company cannot tell whether the journey adds sales, bring MAKINAI at https://makinai.co/services/en/crm-ecommerce-commerce-transformation one journey, its entry rule, overlapping exposures, and the revenue event. We can scope the assignment, holdout, and smallest defensible pilot.

Sources and references

  1. A/B tests in Customer Insights - Journeys · Microsoft Learn

    Documents an A/B test as a default experience versus a variant; its initial audience is delayed until a winner is selected and then receives that winner, so it is not automatically a no-journey group.

    2026-11-14
  2. Set a business goal and measure progress through a journey · Microsoft Learn

    Shows that journey goals observe actions such as purchase, registration, or renewal under product-specific counting windows; observing a goal does not itself establish incrementality.

    2026-11-14
  3. Define your journey's properties — measure lift with a holdout group · Adobe Experience League

    Defines a journey holdout as a share excluded from all journey communication so outcomes can be compared for incremental lift, while documenting availability, sample, and cross-journey limits.

    2026-11-14
  4. Path Experiment Element · Salesforce Help

    Documents random assignment of people to journey paths and explains that realized proportions can differ from configured targets, particularly for small audiences.

    2026-11-14
  5. Completely randomized designs · NIST/SEMATECH e-Handbook of Statistical Methods

    Explains that randomly assigning experimental units to factor levels supports a defensible comparison of that factor under the design assumptions.

    2026-11-14
Making connections

Continue exploring

CRM, Lifecycle & Personalization

Your CRM chooses email, phone, or messaging by habit. Which channel creates conversations?

Read insight ↗
CRM, Lifecycle & Personalization

The form captured a lead. Did it reach CRM? Build the delivery layer before AI

Read insight ↗
CRM, Lifecycle & Personalization

Your best leads arrive when sales is offline. What should AI do until the next shift?

Read insight ↗
Related capability

CRM, commerce & new models

A modern CRM and commerce strategy connects identity, data, communication, experience, offer and operations. MAKINAI helps companies organize that architecture and implement journeys that learn from customer behavior, coordinate channels and prepare the business for agents and emerging buying interfaces.

Explore this capability ↗