Controlling the Cost of AI Features in SaaS Products

Measure a completed workflow, not an API call

Token prices make AI costs look easy to forecast: multiply input and output tokens by a published rate. That calculation is necessary, but it does not describe the cost of the feature. One user action may retrieve documents, rerank them, call a model several times, invoke tools, retry failures and send the result through another evaluator.

The useful unit is cost per successful business outcome: an invoice correctly processed, a case summary accepted or a support request resolved. Include model calls, embeddings, search, storage, moderation, orchestration and human review. A cheap response that is rejected and regenerated three times is more expensive than a higher-priced response that completes the task once.

Link economics to quality from the beginning. Otherwise cost optimisation rewards shorter, cheaper outputs while quietly increasing corrections, escalations or customer harm. The target is the least expensive configuration that meets a defined quality and service threshold-not the lowest invoice from the model provider.

Understand where the bill actually comes from

Input and output tokens are often priced differently, and output generation usually contributes more latency. Reasoning, image, audio, fine-tuned and hosted models may use different meters. Retrieval adds embedding, indexing and query costs. Provisioned deployments charge for reserved capacity even when idle. Pricing pages change, so keep rates in configuration or a cost catalogue rather than embedding them in application logic.

Build a cost model at request level:

workflow cost = inference + retrieval + tools + retries + evaluation + storage + review

Estimate the p50, p95 and worst permitted case, not just the average. Long documents, extended conversations and agent loops create heavy-tailed consumption. The average can look healthy while a small number of requests determines the monthly bill.

Invoice data remains the financial source of truth, but provider billing often arrives too late for request-time control. Record usage returned by each model call and reconcile those estimates with billed meter data. Differences can reveal missing workloads, incorrect pricing assumptions or telemetry that excludes failed and retried work.

Context growth is the cost leak most teams discover late

Chat interfaces encourage developers to resend the entire conversation on every turn. That makes cost grow with the sum of all previous turns, not merely with the new message. A long session can therefore become progressively slower and more expensive even when the user's questions remain short.

Set an explicit context budget. Retain authoritative facts and recent interaction, summarise older material where the quality trade-off is tested, and discard irrelevant tool results. Do not let a model decide which confidential instructions are safe to forget. Conversation compaction is part of application state management.

Retrieval can produce the same problem. Increasing the number or size of chunks may improve the chance of finding evidence but can dilute relevance and increase input cost. Measure retrieval recall and answer quality as context changes. “Send everything” is neither a security strategy nor a cost strategy.

Cap output according to the task. A classifier needs a small schema, not an essay allowance. A generous maximum output can also consume rate-limit capacity even when typical answers are short, depending on how a provider performs admission control.

Route by difficulty, not by user enthusiasm

The largest model is not automatically the most economical. Evaluate candidate models on representative work and identify where smaller models, deterministic code or no AI at all meet the requirement. Use the expensive model only where measured quality justifies it.

Routing can be based on task type, risk, input complexity or the result of a cheaper first pass. However, a router is another model or rule set to evaluate. A cheap classifier that sends difficult cases down the wrong path can reduce both quality and savings.

Prefer deterministic routing where the distinction is already known: document type, language, tenant plan or workflow stage. If escalation follows low confidence, calibrate the signal against actual outcomes. A model saying it is confident is not evidence that it is correct.

Track the effective cost of each route, including failed first attempts. A cascade that calls a small model and then a large model for most requests costs more than calling the large model directly.

Caching is safe only when identity and freshness are in the key

Prompt or response caching can reduce repeated input processing, but business software rarely has globally interchangeable responses. The result may depend on tenant, user permissions, source-document versions, prompt version, model configuration, locale and time. Omitting any of those dimensions can return stale or unauthorised content.

Cache deterministic intermediate work first: parsed document text, embeddings for unchanged content, stable retrieval results and policy lookups. They are easier to invalidate than a complete generated answer. Where a provider offers prefix caching, structure prompts so genuinely stable instructions appear before variable content, but verify the provider's eligibility and privacy semantics.

Semantic caching-reusing an answer for a merely similar question-is higher risk. Similar wording can conceal a different customer, reporting period or permission boundary. Use it for low-risk public knowledge only after measuring false matches. Never make cross-tenant reuse the default.

A cache hit is not free if validation, lookup or storage costs dominate. Measure net savings and operational complexity, including invalidation failures. Some workloads are cheaper to recompute than to cache safely.

Batching trades immediacy for economics

Offline jobs such as catalogue enrichment, historical classification or nightly summarisation need not compete with interactive traffic. Batch facilities may offer different pricing or throughput characteristics and let the provider schedule work more efficiently. They also change failure and support behaviour.

Give every item a stable identifier and make result ingestion idempotent. A batch may partially succeed, expire or return items out of order. Retry failed items rather than the entire file, and preserve the evaluated model and prompt version so a long-running job does not produce an unexplained mixture of behaviours.

Batching is unsuitable when a user is waiting or when source data may become invalid before completion. Define a freshness deadline and recheck permissions and record versions before applying results. A discounted answer based on revoked access is not a saving.

Retries and agents can multiply cost invisibly

A timeout does not prove the provider stopped processing the request. Retrying may pay twice for one intended result. Honour provider retry guidance, use bounded exponential back-off for transient errors and record an operation identifier. Do not retry safety refusals, invalid requests or context-limit failures without changing the request.

Agent loops are more dangerous because the multiplier is inside the workflow. A model can search repeatedly, call the same tool with minor variations or ask itself to reconsider. Set maximum model calls, tool calls, tokens, elapsed time and external spend for a single operation. Stop when progress stalls, not merely when a large global limit is reached.

Cancellation should propagate to queued work and tool execution. Closing a browser tab is not a cost control if background orchestration continues for ten minutes. Record why a loop ended so product teams can distinguish successful completion, budget exhaustion and defective planning.

Tenant limits protect margin and availability

A provider quota controls a deployment or subscription; it does not express the commercial entitlement of an individual SaaS customer. Without application-level allocation, one tenant can consume the shared budget and rate limit, degrading service for everyone else.

Attribute every request to a trusted tenant identifier and feature before it enters a queue. Maintain separate limits for request count, tokens, concurrency and expensive workflow operations. Tokens alone are insufficient when image processing, tools or human review drive cost.

Use several enforcement horizons:

• Per request: maximum context, output, calls and elapsed time.
• Short window: rate and concurrency limits to protect capacity.
• Billing period: included usage, warnings, overage or suspension policy.
• Platform-wide: emergency ceilings that prevent a defect becoming an unlimited bill.

Make the user experience explicit. A silent downgrade produces confusing quality changes; an abrupt refusal can block essential work. Show remaining allowance where appropriate, degrade non-essential features first and preserve the non-AI path.

Rate limits are not spending budgets

Teams sometimes treat tokens-per-minute quota as a financial ceiling. It is primarily a capacity and admission-control mechanism. Providers may estimate request size using configured maximum output, and optional dynamic capacity can process additional work when available. Neither behaviour guarantees a monthly spending limit.

Likewise, a 429 response does not always mean the monthly allowance is exhausted. Bursty traffic, provider capacity or estimated token reservations can trigger throttling. Monitor response headers and billed usage separately, queue non-interactive work and avoid aggressive retries that turn throttling into more load.

Application budgets must reject or defer work before calling the model. Cloud budget alerts remain useful for finance and incident response, but delayed alerts cannot stop a fast loop. Put the hard control close to the operation that creates cost.

Choose consumption or provisioned capacity from measured demand

Do not buy a commitment from a spreadsheet forecast alone. Load-test the evaluated workload, including its real prompt and output distributions, and confirm capacity availability before purchasing a reservation. Revisit utilisation because model routing and prompt optimisation can reduce demand after the commitment is made.

Cost observability needs product dimensions

A cloud invoice can identify a resource but usually cannot explain which feature, customer or release created the spend. Emit structured usage events containing tenant, feature, workflow, model deployment, prompt version, input and output usage, cache status, retry number and outcome. Avoid storing sensitive prompt content merely to calculate cost.

Dashboards should answer:

• What is the cost per successful outcome and per active tenant?
• Which workflows have the highest p95 and maximum cost?
• How much spend comes from retries, failed work and evaluation?
• Which model or prompt release changed cost or quality?
• Are included allowances and subscription prices covering marginal cost?

Alert on rates and anomalies, not only monthly totals. A sudden rise in tokens per workflow, cache misses or loop termination can identify a regression while the absolute bill is still small.

Optimise in the right order

1. Remove unnecessary calls. Use rules, search or existing data when the answer is deterministic.
2. Bound the workflow. Cap context, output, iterations, concurrency and tenant usage.
3. Improve first-pass success. Better retrieval and validation often save more than a cheaper model.
4. Route evaluated tasks. Use smaller models where they demonstrably meet the quality floor.
5. Reuse safe work. Cache stable, correctly scoped intermediates and batch delay-tolerant jobs.
6. Optimise the purchasing model. Consider commitments only after demand is understood.

Re-run quality evaluation after each optimisation. Reducing context, changing models or compressing history is a behavioural change, even when the code path looks like a performance improvement.

Related C3 Software guides

• How to Add AI Features to Existing Business Software Safely
• Evaluating AI Output Quality in Production Software
• Using AI for Document Processing Without Losing Human Oversight
• Designing Resilient .NET Integrations with External APIs