Why AI Projects Stall Between Demo and Production
The few steps between a working demo and a live system — and how to plan for them up front.
Define how work continues when the service fails
A demonstration follows prepared inputs through one successful path. A live service also encounters timeouts, congestion, repeated clicks, cancellation and document changes. Include these in business acceptance before promising a launch schedule. Define users, permitted actions, human handoffs and who may pause the service.
Hypothetical illustration, not a real customer case or capacity commitment. Suppose an internal assistant searches a knowledge base and creates a support ticket, expanding from
5pilot users to50. Logical tasktask-42creates a record in the ticketing system, but the return connection breaks. The user sees a timeout, clicks again, then cancels. Treating every click as a new task can create duplicate tickets. Disabling a cancel button does not establish that the external record was undone.
Use the failed task to expose dependencies
Map identity, the knowledge index, the model, the task queue and ticketing. Under the same task identifier, record queue time, dependency durations, retries, versions and final state while minimising raw customer content in logs. Locate where progress stopped before changing the model, connection, quota or workflow.
| Failed dependency | Diagnostic question | How business work continues |
|---|---|---|
| Identity or permissions | Is identity still verifiable, and is access current? | Stop protected operations when access cannot be confirmed |
| Index update | Do source and index versions match? | Pause answers depending on new material; provide human verification |
| Model throttling or outage | Are quota, concurrency, waiting and retries compounding? | Bound queuing/backoff and hand off after the task deadline |
| Unknown ticket-write result | Was an external record already created? | Reconcile state before creating again |
| Demand backlog | How old is the queue, what is in flight, and which dependency is slow? | Limit, reject or defer new work rather than queue without bounds |
Moving from 5 to 50 users is not a simple multiplication of model capacity. Tasks may call several services, and retries consume resources too. Exercise representative concurrency and slow dependencies to inspect queuing, cancellation, cost and failure. Set queue and task limits before widening access.
Retries, duplicate requests and cancellation differ
Give a logical intent a stable task identifier. Reuse its idempotency key on retries and retain a request fingerprint and result state. Do not deduplicate solely by prompt text, which could suppress legitimate similar requests. If the external system supports idempotency, check its scope and lifetime. Otherwise reduce duplicate risk through reconciliation, controlled serialisation or human review; do not promise unconditional exactly-once execution.
A timeout means the client did not receive a timely result, not that the external action never occurred. For task-42, query the created record before returning the original result or arranging compensation. See the AWS discussion of timeouts and retries for bounded retries and backoff considerations.
Cancellation may mean stopping the wait, stopping later steps, requesting upstream cancellation or undoing a completed action. These are separate operations. After recording cancel_requested, still handle late results. If completion or cancellation cannot be confirmed, record unknown and send the task for reconciliation. An existing ticket needs an explicit closure or compensation process; the screen shown after cancellation must not imply that no write occurred.
Bound updates and rollback
Version code, models, prompts, indexes, permission rules and tool schemas. After the document owner approves an update, check ingestion, cache invalidation and citation access, then rerun affected samples. Rolling back code must not restore revoked access or expired policies. Disable the affected function if necessary.
Review fallback models for processing location, charges, tool parameters and answer behaviour. Without an approved alternative, provide a retry-later explanation or human handoff. Before reverting a release, define how in-flight work is paused, drained or reconciled. Reverting a page or prompt alone does not resolve external side effects; their owner must handle them separately.
Reusable release gates with named owners
- Business owner: approve scenarios, handoffs and failure messages. Start read-only; add suggestions or writes according to acceptance evidence.
- Data and security owners: approve versions, permissions, logs and processing boundaries. Failed revocation, unauthorised retrieval or unconfirmed writes block expansion.
- Technical owner: rehearse dependency failures, duplicates, cancellation, late callbacks and insufficient capacity; retain task-state and reconciliation evidence.
- Operations owner: document pause conditions, restorable versions, switches and rollback. Assign responsibility for in-flight work and external records.
- Joint acceptance: agree quality, waiting-time, cost and update criteria. Keep the pilot limited while any critical item lacks an owner or reviewable result.
Release follows this evidence and responsibility, not a successful demonstration or a fixed number of days. Gradual expansion gives the team room to correct the workflow while the impact of failures remains manageable.