Direct answer: do not train or evaluate lead scoring on qualified versus disqualified, or converted versus not converted, without first recording who was eligible, prioritized, offered service, actually worked, and allowed to mature. A lead that never received meaningful follow-up is not a negative label; its outcome is unobserved. Create a bounded exploration cohort within commercial guardrails so eligible leads outside the top score receive comparable service and reveal what the historical queue could not observe.
The loop can look successful. AI places certain profiles at the top. Sellers call them first, attempt more often, and use the strongest channel. Those leads convert more. At retraining, the system concludes those profiles were superior and raises their priority again. Similar records lower in the queue receive a late attempt or none, so the company never learns whether they would have bought with the same service. Accuracy can improve within the worked population while potential revenue remains invisible.
Decision research calls this a selective-label problem: an outcome becomes observable only where a prior policy allowed observation. The cited NBER paper studies judicial decisions, not selling, so its effect estimates do not transfer to marketing. The structure does: evaluating a new policy with data generated by the old policy requires separating underlying potential from prior human selection.
Stop treating no observed outcome as a loss
- Eligible — the lead and offer could enter the process at that time.
- Prioritized — the policy surfaced or offered the record to a seller.
- Accepted — an owner assumed responsibility within the defined window.
- Served — a valid attempt occurred through the required channel, timing, and cadence.
- Observable — instrumentation and the maturity window can establish what happened.
- Outcome — conversation, opportunity, sale, revenue, or loss with source and reason.
“No conversion” can mean correct service without a purchase, insufficient service, a logging failure, or no service at all. Compressing all four into zero teaches the model the current operation rather than demand. The guide at https://makinai.co/insights/en/offer-eligibility-before-ai-lead-scoring determines who may receive the offer; https://makinai.co/insights/en/ai-lead-queue-orchestration-score-sla-capacity determines who should be served next. This guide determines which observed outcomes are fit to become training labels.
Write a Sales Exposure Contract
- Unit — one lead eligible for one offer and one defined commercial action.
- Policy — score, queue, territory, owner, capacity, and version that determined priority.
- Exposure — offer, acceptance, and service timestamps plus channel, business hours, attempt, and SLA.
- Intensity — minimum comparable service quality without turning contact volume into pressure.
- Observability — fields, integrations, and window needed to distinguish loss, absence, and unknown.
- Outcome — versioned definition, source, value, maturity, and any later correction.
- Exploration — safe population outside the top band, selection rule, reserved capacity, and limits.
- Abstention — records without sufficient exposure do not become negatives in the primary training set.
- Audit — recommendation, human decision, service delivered, override, and outcome stay connected.
Microsoft documents predictive models trained on historical qualified and disqualified leads, using configurable filters and periods. Salesforce documents patterns learned from conversion history, fields, and activities. These are useful capabilities, but historical states may encode territory, capacity, shift, channel, language, or seller preference. The key question is not only whether the model uses history; it is which operating policy generated that history.
Measure the denominator the model cannot see
AUC, F1, recall, and confusion matrices help on labeled records. They cannot by themselves show what would happen if unworked leads received comparable service. Measure label coverage over all eligible records; offer, acceptance, and service rates by score band; valid attempts; response time; unknown outcomes; conversion conditional on service; matured revenue; and service concentration by source, region, segment, and language.
Build a Potential–Service Matrix with four cells: high potential/high service, high potential/low service, low potential/high service, and low potential/low service. The low-service cells expose an evidence gap; they do not prove missed conversion. Reviewed samples and controlled exposure turn some of that gap into evidence.
In the United States, outsourced SDRs, territory rules, inbound versus outbound teams, call systems, and partner channels often produce different service intensity. A model can confuse the operating model with customer propensity. Privacy, telemarketing, consumer protection, and sector requirements remain hard gates; exploration never overrides permission, eligibility, or do-not-contact rules.
Run a six-week pilot on one offer
- Week 1 — extract eight to twelve weeks of all eligible leads, not only closed records; rebuild priority, owner, attempts, and outcomes.
- Week 2 — define comparable minimum service, clocks, observability, missing-outcome reasons, and cohorts excluded from exploration.
- Week 3 — run the contract in shadow mode and quantify where score and service are confounded.
- Week 4 — reserve bounded capacity for an eligible sample outside the top band with the same SLA and controls; do not remove service from urgent records.
- Week 5 — review contact quality, workload, exceptions, and early outcomes; repair logging before retraining.
- Week 6 — compare coverage, conditional conversion, matured revenue, and cost; retain rules, recalibrate, or collect more evidence.
Deliverables include an exposure map, state dictionary, cohort baseline, exploration policy, instrumentation, reviewed sample, dashboard, runbook, and retraining decision. Major costs are usually activity capture outside the CRM, reason-code cleanup, telephony and messaging integration, reserved seller capacity, monitoring, analysis, and incentive change—not the model license.
Protect revenue and operations together
Countermetrics include slower SLA for urgent demand, seller overload, inappropriate contact, uneven service distribution, unsubscribe growth, underpowered exploration, and labels corrected after close. NIST recommends documenting selection, representativeness, knowledge limits, and evaluation in deployment-like conditions. Preserve human review, but record overrides as data: if sellers consistently ignore a band, capture the reason rather than letting silence become truth.
Use https://makinai.co/insights/en/crm-native-ai-lead-scoring-vs-custom-model-investment to choose rules, native scoring, or a custom model, and https://makinai.co/insights/en/ai-lead-scoring-model-drift-pipeline-quality to monitor drift after launch. If conversion improves as service becomes more concentrated, bring MAKINAI one offer and an anonymized sample of every eligible lead with service timestamps through https://makinai.co/services/en/crm-ecommerce-commerce-transformation. We can scope the contract, instrumentation, and pilot before the next retraining cycle.
The sources establish mechanisms and risk; they do not promise lift. The aggregate 28-day snapshot ending November 6 recorded 180 sessions, 11 organic clicks, and zero lead events. It cannot attribute inquiries, pipeline, or revenue to this article or validate the hypothesis for a specific company.