Direct answer: hire the provider that can connect every bias claim to a specific decision, affected groups, data provenance, suitable metrics, and the consequences of intervention. Require reproducible results, uncertainty, performance trade-offs, acceptance criteria, and a monitoring plan. A report with one fairness score but no counts, denominators, context, or artifacts is not enough.
Start by defining what is being evaluated: lending, applicant screening, recommendations, customer service, fraud, moderation, or another decision. Data bias, measurement error, uneven coverage, proxy effects, business rules, and operating behavior require different remedies. NIST's AI RMF calls for fairness and bias to be evaluated and documented, while warning that mitigating harmful bias does not by itself make a system fair. A credible provider explains the boundary of the assessment instead of promising neutrality.
Bias Mitigation Provider Proof-8: a 32-point scorecard
Score each proof from zero to four: zero is absent; one is an assertion; two is a checkable description; three is reproducible evidence; four adds a contextual test and operating owner. As an initial gate, require 24 of 32, no zero, and at least three points for scope, metrics, reproducibility, and monitoring. Adapt weights to the decision's risk.
- Decision scope — outcome, user, timing, automated action, human review, plausible harm, and out-of-scope use are documented.
- Groups and participation — relevant groups, intersections, accessibility, affected people, and sample limitations are justified.
- Data lineage — source, permitted use, coverage, period, labels, proxies, missingness, and distribution changes are auditable.
- Metrics and uncertainty — selection rates, errors, quality of service, calibration, and case-appropriate measures include counts, denominators, and uncertainty.
- Causal and operational diagnosis — the team distinguishes correlation from cause and tests data, objective, model, threshold, interface, process, and human intervention.
- Mitigation alternatives — preprocessing, training, post-processing, rules, redesign, and human controls are compared with performance loss and harm displacement.
- Independence and reproducibility — code, versions, parameters, decisions, and conflicts support review by another qualified team.
- Operations and remedy — monitoring, triggers, incidents, contestability, correction, retirement, and post-launch owners are defined.
Six disqualifying gates
- The provider promises to remove bias without defining fairness, context, or affected groups.
- One score summarizes the system without sample size, denominators, uncertainty, or subgroup detail.
- The same company audits its own tool without disclosing incentives, access constraints, or an independent safeguard.
- Protected attributes or proxies are used without applicable purpose, governance, security, and legal review.
- A mitigation improves one measure without retesting utility, other groups, and the operating process.
- No reproducibility, drift monitoring, contestability channel, or accountable owner exists for correction or suspension.
Run a paid reproducibility challenge
Give finalists the same authorized, minimized package: decision, controlled dataset or sample, dictionary, policy, baseline, and questions. Ask each team for a scope record, coverage analysis, subgroup metrics, uncertainty, causal hypotheses, two mitigation alternatives, and a production plan. Add one threshold or sample-composition change to see whether the recommendation remains stable.
Pay for the exercise and do not share unnecessary personal data. Synthetic data can test method, but it cannot prove how the real system behaves. Compare limits, reproducibility, contradiction-finding, and willingness to recommend process change or suspension—not only technical tuning.
Turn the report into acceptance criteria
Before award, lock the protocol, evaluation sets and periods, groups and intersections, metrics, minimum sample, low-power treatment, thresholds, quality tests, baseline, artifacts, human review, and permitted decisions. Require tables with counts as well as percentages, and record who accepts each trade-off.
For employment uses in the United States, involve counsel and domain specialists. The EEOC states that anti-discrimination laws still apply when AI is used in employment decisions. New York City's rule for covered automated employment decision tools adds specific audit, publication, and notice requirements. Do not generalize those duties to every system or jurisdiction. A technical provider should map evidence to applicable advice, not impersonate legal counsel.
Use a six-week implementation
Week 1: scope the decision, users, jurisdictions, and harms. Week 2: map data lineage and subgroup coverage. Week 3: reproduce the baseline and measure uncertainty. Week 4: test causes and mitigation alternatives. Week 5: validate trade-offs with business, risk, legal, and affected-user perspectives. Week 6: publish the decision, owners, limits, monitoring, and pause triggers in shadow mode before changing production.
The work does not end with the report. Monitor changes in population, channel, model, threshold, interface, data, policy, and human operators. Separate investigation alerts from remediation and suspension. Track incidents, complaints, overrides, and groups receiving weaker quality of service. A credible provider says when evidence is insufficient and what additional data would be proportionate.
Connect bias review to the full buying decision
Evaluate the team in https://makinai.co/insights/en/how-to-evaluate-ai-consulting-team-before-hiring, normalize the proposal in https://makinai.co/insights/en/how-to-evaluate-ai-consulting-proposals-scorecard, and expose platform incentives in https://makinai.co/insights/en/assess-vendor-neutral-platform-aligned-ai-consultancy. If you need to turn a fairness concern into a protocol, paid test, acceptance criteria, and operating controls, bring MAKINAI at https://makinai.co/services/en/ai-strategy-transformation-consulting the decision, data map, relevant groups, jurisdiction, and any known incident. The first deliverable should be a defensible scope—not a promise of “bias-free AI.”