LLM Cost Governance: What to Actually Measure

Measure the cost of an accepted business result

A document-answering task can include retrieval, generation, tool validation, retries and human correction. Define completion first: which checks must pass, who handles failures, and how cancellation or timeouts are counted. Group costs by business task so the comparison buys an equivalent usable result. A successful provider request does not establish business acceptance.

A total-cost formula for one task

C_task = Σ C_model_attempt + C_retrieval + C_tools + C_runtime + C_human
C_model_attempt = (U×p_in + R×p_read + O×p_out + W×p_write) / 1,000,000
C_human = minutes / 60 × hourly_rate
C_per_accepted = batch_total_cost / accepted_task_count

U is uncached input, R cached reads, O output and W separately charged cache writes, all in tokens. Each p is an illustrative rate in USD per million tokens. Use one currency and batch boundary. Include only applicable billing categories: if writes are already covered by input pricing, do not add them again. Retrieval, tools, runtime and labor also need nonoverlapping allocation rules.

Failed attempts remain in the numerator; only accepted tasks enter the denominator. A missing response is not evidence of zero consumption: reconcile provider records first. Allocate shared cache and runtime charges by a fixed, reviewable rule.

Illustrative assumptions: a batch of 100 tasks

Every number below is an illustrative assumption. This is not a real customer case, a cloud/model quotation or a savings commitment.

• There are 100 tasks. A full attempt consumes 2,000 input and 500 output tokens. Failure here means business validation fails after a complete output, so unsuccessful attempts incur those same token charges.

• The primary model has fictional input/output rates of 2/8 USD per million tokens: 0.004 + 0.004 = 0.008 USD per attempt. Retrieval happens once per task at 0.001 USD and retries reuse the retrieved material. Validation/tools cost 0.0005 USD per attempt. Allocated batch runtime is fixed at 0.080 USD. Labor is zero in the comparison table.

• The baseline has 20 first-pass failures, each retried once. For this calculation only, assume every retry is accepted and all 100 tasks complete. Real evaluation must retain terminal failures rather than assume this outcome.

The complete calculation

Caching

A hit reuses 1,000 input tokens at a fictional read rate of 0.5 USD per million; the other 1,000 input tokens remain normally priced. A hit therefore costs 0.002 + 0.0005 + 0.004 = 0.0065 USD. Of 120 attempts, 10 are cold and 110 hit. Ten writes each store 1,000 tokens at an assumed additional write rate of 1 USD per million. Content and permissions are unchanged; first-pass failures remain 20.

Shorter prompts

First attempts use 1,000 input and 500 output tokens, costing 0.006 USD. Assume missing constraints increase first-pass failures to 40. Each repair restores the full 2,000-token input and costs 0.008 USD. Repairs are not charged at the shortened-prompt rate.

Model routing

Route 40 simple tasks to a small model with fictional input/output rates of 0.5/2 USD per million, or 0.002 USD per attempt. Sixteen fail validation and escalate to the primary model. The other 60 tasks start on the primary model and 12 need a retry. Primary attempts total 60 + 16 + 12 = 88. There are 28 first-pass failures and 128 attempts overall.

Baseline: 120 × 0.008 = 0.960
          0.960 + 0.100 + 0.060 + 0.080 = 1.200
Cache:    10 × 0.008 + 110 × 0.0065 = 0.795
          writes = 10 × 1,000 × 1 / 1,000,000 = 0.010
          0.795 + 0.010 + 0.100 + 0.060 + 0.080 = 1.045
Short:    100 × 0.006 + 40 × 0.008 = 0.920
          0.920 + 0.100 + 0.070 + 0.080 = 1.170
Routing:  40 × 0.002 + (60 + 16 + 12) × 0.008 = 0.784
          0.784 + 0.100 + 0.064 + 0.080 = 1.028

Approach	First-pass failures /100	Attempts	Batch cost USD
Baseline	20	120	1.200
Caching	20	120	1.045
Short prompt	40	140	1.170
Model routing	28	128	1.028

Approach	Cost / accepted task USD
Baseline	0.01200
Caching	0.01045
Short prompt	0.01170
Model routing	0.01028

Consider a separate terminal-failure case: if the baseline still costs 1.200 USD but only 95 tasks are accepted, the cost per accepted task is 1.200 ÷ 95 ≈ 0.01263 USD. If the other five each need two minutes of labor at a fictional 30 USD/hour, that adds 5 × 2 ÷ 60 × 30 = 5 USD. Assuming all five are then repaired, cost per completed task becomes (1.200 + 5) ÷ 100 = 0.062 USD. These additional assumptions are not included in the table.

Choose with quality and failure paths in view

Caching lowers the illustrative total without changing first-pass failures only if document versions, permissions, expiry and cache keys are correct. Lower hit rates, frequent writes or stale hits change the result. Measure hit rate and quality rather than read price alone.

Shortening halves input per first-pass request, but batch input changes from 120 × 2,000 = 240,000 to 100 × 1,000 + 40 × 2,000 = 180,000 tokens, only 25% less. Repairs offset part of the reduction; batch cost changes only from 1.200 to 1.170, while first-pass failures rise from 20 to 40. Revert if latency or correction work is unacceptable even when token use falls.

Routing lowers this example cost but raises first-pass failures to 28. It is an application strategy requiring explicit customer approval for multiple models; it does not mean Corebyte substitutes a purchased model. Never silently downgrade. Record the actual model, routing reason and fallback, and test classification and escalation on the same samples including boundary cases.

When usage and invoices disagree

1. Align the boundary: account/project, exact model and version, region and route, currency, billing period and time zone. Separate provisional usage estimates from settled invoices.

2. Join every attempt and provider request identifier to a task. Check SDK retries, application retries, resend-after-timeout and interrupted streams. One click can create several billable attempts. Mark unknown outcomes for reconciliation rather than adding or dropping them twice.

3. Align units and definitions: input, output, cache reads/writes and any reasoning/tool categories must follow provider fields. Do not add cached input again if already included in total input. Convert tokens, million tokens, seconds and capacity-hours separately.

4. Aggregate requests into the billing period, then reconcile delayed posting, package credits, discounts/refunds, taxes and separately agreed service fees. Retain unexplained differences instead of inventing an effective unit rate.

5. Investigate largest differences first: unattributed charges, duplicates, usage categories and rate conditions. Keep redacted request identifiers, usage fields, timestamps and invoice rows; do not retain credentials, full prompts or customer documents by default.

A reusable reconciliation checklist

• Each record has task_id, attempt_id, provider request ID, model/version, route, start/end, technical status and business acceptance.
• Preserve original units; separate model, retrieval, tool, runtime and labor costs with their sources and allocation rules.
• Compare cost, acceptance, retries, end-to-end latency and corrections on the same sample set, not just cost per call.
• Bound output, loops, retries and concurrency; define quota exhaustion, timeout and unknown-result handling. A cost alert is not a rate limiter.
• Reconcile again after model, cache or workflow changes. Do not promise a fixed saving without sample and billing evidence.

For real prices and categories, consult Amazon Bedrock pricing or the current terms of your chosen provider. The fictional data here demonstrates a calculation and diagnostic method only.