Direct answer: assign three groups before the first message. A and B receive different versions of the same journey; an eligible holdout does not enter that journey during the defined window. A versus B selects a treatment. A or B versus holdout estimates the journey's effect. If a group called control receives the default version or merely waits to receive the winner, it does not answer the second question.
Platform vocabulary can hide the distinction. Customer Insights calls version A a control and documents an initial test audience that can receive the winner later. Salesforce randomizes journey paths. Adobe's documented journey holdout excludes profiles from all journey communication and compares outcomes, while also noting availability, sample, and cross-journey limits. Put the design in the experiment contract, not only in a platform setting.
Use a three-way journey proof
- Frozen eligibility — who could enter, when, under which product, market, channel, and exclusion rules.
- Stable assignment — person or account ID, experiment version, A, B, or holdout, and a timestamp before exposure.
- Treatment A — the current journey, including message, timing, channel, offer, cost, and policy.
- Treatment B — one bounded change; do not change content, channel, audience, and incentive together.
- No-journey holdout — service and transactional messages continue, but the tested intervention does not.
- Actual exposure — sends, deliveries, failures, other journeys, media, and sales contact.
- Mature outcome — purchase, opportunity, renewal, revenue, or contribution with a window, corrections, and eligible denominator.
- Decision — promote A or B, keep the current rule, redesign, or stop, with an owner and date.
Assign at person level when treatment and contamination are individual. For ABM, franchises, or shared purchases, use account, store, or territory when people in the same unit must not receive conflicting treatments. Randomization does not guarantee equal counts; NIST and platform documentation treat realized allocation as probabilistic. Stratify only on variables chosen before the test and report imbalance.
Separate four questions that dashboards blend
- Execution — did the journey enter, deliver, and exit as designed?
- Optimization — did B outperform A on the selected outcome?
- Incrementality — did treatment outperform eligible people who received no journey?
- Economics — did incremental contribution cover messaging, incentive, media, data, review, and operations?
Clicks and opens help debug delivery, but a no-message holdout cannot compete on opens. Use a downstream outcome compatible with the buying cycle. A native journey goal is operationally useful, yet it may apply its own counting and attribution window. Reconcile the event to CRM, commerce, and, where the decision requires it, the revenue rung in https://makinai.co/insights/en/crm-closed-won-booked-billed-collected-revenue-ai.
Illustrative U.S. example
A hypothetical midsize subscription company has 30,000 customers eligible for a second-purchase journey. It assigns 12,000 to the current rule, 12,000 to an AI-assisted recommendation, and 6,000 to no journey. After the purchase and return window, A converts at 5.0%, B at 5.4%, and holdout at 4.8%. B beat A by 0.4 percentage point, but its difference versus holdout is 0.6 point; A added 0.2 point. Only then should the team apply contribution, discounts, contact cost, and uncertainty. This is an illustration, not a client case or benchmark.
If holdout received a parallel promotion or some customers in B never received the message, analyze by original assignment and show actual exposure separately. Removing undelivered records after randomization favors treatment. The primary analysis should keep all eligible units in their assigned arm; exposure-based views are diagnostic.
Implement in six weeks
- Week 1 — select one frequent, reversible journey; define unit, eligibility, intervention, outcome, window, and smallest commercially useful effect.
- Week 2 — map other journeys, sales, media, consent, transactional messages, exclusions, and the authoritative outcome source.
- Week 3 — implement persistent assignment plus entry, exposure, exclusion, outcome, and correction events; test duplication and reentry.
- Week 4 — run shadow assignment while the current journey remains unchanged; reconcile counts and identities.
- Week 5 — release A, B, and holdout at approved volume; monitor safety, contact pressure, delivery failure, and contamination.
- Week 6 — read only the first available window; do not declare a winner before revenue or returns mature; lock the final read date.
Size investment and trade-offs
Cost rises with journey and channel count, cross-device identity, assignment unit, reentry, duration, late outcomes, offline purchase, returns, consent, external data, campaign overlap, minimum volume, and the need for a global holdout. A journey-level holdout is simpler and answers one intervention. A persistent cross-journey holdout estimates the portfolio but requires shared orchestration and withholds more treatment.
Do not promise a universal holdout percentage. Size and duration depend on baseline conversion, variance, minimum useful effect, unit, clustering, outcome maturity, and the cost of a wrong decision. Small populations may support only A versus the current rule, a sequence of tests, or a quasi-experiment. An inconclusive result is evidence; it is not permission to select the prettier version.
Define rules, AI, and accountability
Rules own eligibility, consent, assignment, exclusions, transactional classes, frequency, windows, and stopping. AI can develop variants, estimate heterogeneity after the test, detect contamination, and propose hypotheses. It should not change arms, outcome, or window mid-test, remove unfavorable cases, or promote a click winner when the decision is revenue. Marketing owns the hypothesis; CRM/MarTech, delivery; Data, randomization and analysis; Sales and Service, parallel contact; Finance, mature value; leadership, holdout cost and release.
Release value, not just a winner
- Eligible coverage, stable assignment, baseline balance, and improper reentry.
- Delivery and contamination by arm, channel, journey, and sales contact.
- B minus A and treated minus holdout, with uncertainty and absolute effect size.
- Incremental revenue, contribution, and cost per eligible unit after returns and maturity.
- Opt-outs, complaints, contact pressure, service cases, and approved segment effects.
- Time to decision, experiment cost, and decisions changed by the evidence.
Coordinate the portfolio through https://makinai.co/insights/en/ai-marketing-capacity-valid-experiment-operating-system, reconcile cross-channel exposure with https://makinai.co/insights/en/ai-cross-channel-contact-pressure-crm-frequency, and mature the financial outcome through https://makinai.co/insights/en/crm-closed-won-booked-billed-collected-revenue-ai. If A/B tests select messages but the company cannot tell whether the journey adds sales, bring MAKINAI at https://makinai.co/services/en/crm-ecommerce-commerce-transformation one journey, its entry rule, overlapping exposures, and the revenue event. We can scope the assignment, holdout, and smallest defensible pilot.