The direct answer: do not scale a pilot because the demo impressed stakeholders or a few outputs looked good. Move it into production only when four conditions are demonstrated together: the workflow improves a meaningful marketing decision or deliverable; quality and risk are tested on real cases; operators can detect, review, and reverse failures; and cost per useful unit remains acceptable at expected volume. The transition deliverable is not a model. It is an operating system for the workflow.
During a pilot, the best operator selects examples, repairs inputs, and works around failures. Production brings incomplete briefs, stale documents, volume spikes, new markets, model changes, and people who lack project context. Repeating the prompt at scale simply amplifies variability. The company must define what can change, who authorizes it, and how the service returns to a safe state.
Write the Workflow Operating Contract
- Outcome — the marketing decision or deliverable that must improve and its baseline.
- Scope — users, markets, channels, data, volume, and permitted actions.
- Release unit — prompt, rule, source, model, integration, and interface versioned together.
- Evaluation — case set, criteria, approval thresholds, and human review.
- Service level — availability, latency, freshness, capacity, and support window.
- Exception — human queue, fallback, and behavior when data, model, or destination fails.
- Telemetry — observable input, version, decision, correction, cost, action, and outcome.
- Economics — cost per accepted output, useful decision, or incremental result, plus monthly budget.
- Change — who may edit, test, approve, release, and reverse each component.
- Incident and owner — severity, alert, response, rollback, communication, and end-to-end accountability.
The Contract prevents quality from collapsing into one average. A workflow can score well overall while failing on claims, pricing, consent, language, or the highest-value segment. Set hard blocks by critical case class, not only an aggregate grade, and retain real examples for regression testing.
Pass four gates before expanding
- Evidence — the evaluation set represents common, difficult, and critical cases and compares the new release with the current one.
- Operability — logs, alerts, exception queue, fallback, and rollback work with named owners.
- Economics — total cost per accepted unit and human capacity have been measured at projected volume.
- Accountability — marketing, data, technology, agencies, legal, and operations know who decides and who responds when the workflow fails.
OpenAI recommends task-specific evals, logging throughout development, and calibrating automated scoring with human feedback. That does not require automating every judgment. Brand, claim, and new-market cases may remain under review. Evaluation exists to detect regression and authorize change—not to replace an executive decision.
Version the system, not only the prompt
Record the prompt, policy, examples, knowledge source, model, parameters, tools, rules, integrations, and evaluation criteria. Google Cloud recommends version control and CI/CD for the modifiable components of a generative application. Without that system boundary, teams know behavior changed but cannot identify which change caused the regression.
Choose the smallest architecture that can operate
- Native capability — least operating effort, with possible limits in evaluation, portability, and observability.
- Configured workflow — combines rules, models, and integrations with intermediate control.
- Owned layer — justified when the decision differentiates the business or requires specific policies, data, and destinations.
- Hybrid operation — keeps critical decisions with people while automating triage, preparation, or recommendation.
- Real time — only when decision value decays before the next batch and the operation can act inside that window.
Expanding horizontally into more countries and channels before stabilizing one workflow usually multiplies exceptions. Prefer depth: one journey, one user group, and one complete outcome chain. Expand after evaluation, support, and economics demonstrate that they can follow volume.
An eight-week productionization project
- Week 1 — confirm outcome, baseline, scope, and expected volume.
- Week 2 — turn pilot logs and corrections into an evaluation set and critical case classes.
- Week 3 — version components, criteria, and release environments.
- Week 4 — instrument quality, cost, latency, errors, human intervention, and downstream outcomes.
- Week 5 — run in shadow mode and rehearse failures in source, model, integration, and destination.
- Week 6 — release to a bounded cohort with active fallback and rollback.
- Week 7 — compare quality, outcome, unit cost, human load, and incidents.
- Week 8 — expand, revise, remain limited, or stop; record operating owners and budget.
Measure useful service, not model calls
- Business — conversion, revenue, contribution, retention, time, or quality linked to the workflow.
- Quality — acceptance, correction, regression, and critical failure by case class.
- Operations — availability, latency, backlog, overrides, incidents, and recovery time.
- Adoption — eligible usage, abandonment, manual workarounds, and recipient satisfaction.
- Economics — cost per accepted output, useful decision, and incremental result, including people and rework.
- Counter-metrics — error, complaint, brand risk, contact pressure, and a bottleneck shifted to another team.
The FinOps Foundation recommends estimating AI costs from development and pilots through production. Tokens are one line item. Include retrieval, data, storage, environments, evals, observability, review, support, vendors, and incidents. A cheaper model call can cost more if it increases corrections or downstream failure.
Accountability, investment, and limits
Marketing owns the outcome and policy; operations defines service and exceptions; data owns sources and quality; technology integrates, versions, and monitors; the agency or consultancy delivers components and evidence; legal, privacy, and security set boundaries; finance tracks unit cost; leadership authorizes expansion. NIST treats testing before deployment and monitoring during operation as continuous lifecycle activities, not a one-time approval.
In the United States, include vendor change notices, accessibility, industry requirements, after-hours support, and the operating impact of state and sector rules. In every market, the timeline depends on integration, volume, and criticality; eight weeks is a scope boundary, not a guarantee.
Bring the pilot and its failures to the decision
Use the first-workflow plan at https://makinai.co/insights/en/where-start-ai-marketing-90-day-first-workflow, choose autonomy at https://makinai.co/insights/en/copilot-workflow-ai-agent-marketing-what-build-first, organize knowledge at https://makinai.co/insights/en/ai-marketing-copilots-inconsistent-answers-knowledge-layer, and compare buying, configuring, or building at https://makinai.co/insights/en/buy-configure-or-build-ai-solution. MAKINAI structures and implements this transition at https://makinai.co/services/en/ai-strategy-transformation-consulting. Bring one pilot, twenty representative cases, and four weeks of logs, costs, and outcomes; we can define the smallest production scope.
Production does not guarantee value. An evaluation set can age, providers can change models, users can work around the workflow, and a stable process can optimize the wrong metric. The sources support continuous evaluation, versioning, operations, cost estimation, and risk management; the Contract and project design are MAKINAI editorial recommendations to validate in context.