How to Add AI Features to Existing Business Software Safely

Start with the cost of a wrong answer

Adding AI to an existing application is technically easy. Sending text to a model endpoint may take less than an afternoon. The difficult question is what the software will do when the response is plausible, well written and wrong.

That question separates suitable features from dangerous demonstrations. Drafting an internal summary that a user can edit has a different risk profile from approving credit, changing a customer's account or sending a contractual statement. Accuracy alone is not the deciding factor. Reversibility, detectability and the consequence of error matter more.

Classify the proposed use case before choosing a model:

• Assistive: the model drafts, ranks or summarises; a person remains responsible for using the result.
• Advisory: the result influences a decision and must include evidence, uncertainty and a route to challenge it.
• Automated: the result changes state without prior review. This needs deterministic validation, bounded authority and a practical reversal path.

A useful first release is usually assistive. This is not timidity. It lets the team collect representative inputs, corrections and failure examples before automation magnifies errors. If nobody can state the acceptable error rate and the cost of false positives versus false negatives, the feature is not ready to make decisions.

Treat the model as an untrusted subsystem

Developers often describe a model as if it were a new layer of business logic. That mental model is unsafe. A model is closer to an external, non-deterministic service that accepts attacker-influenced input and returns untrusted output. It can fail while returning HTTP 200.

Keep deterministic responsibilities outside the model: authentication, authorisation, tenant isolation, financial calculations, validation and state transitions. The model may propose an action, but ordinary application code must decide whether that action is valid and permitted. A sentence in a system prompt such as “only update records the user may access” is not an access-control mechanism.

This produces a useful boundary:

1. The application authenticates the user and establishes tenant and permissions.
2. Code retrieves only the data that identity is allowed to see.
3. The model works with a deliberately minimised context.
4. Its output is parsed into a typed proposal, not executed as free-form instructions.
5. Application rules validate the proposal and either reject it, request approval or perform a bounded operation.

Structured output improves parsing; it does not prove truth. A perfectly valid JSON object can contain the wrong customer number. Schema validation and semantic validation solve different problems, and production systems need both.

Do not build a second, weaker authorisation system in retrieval

Retrieval-augmented generation creates a subtle security problem. Teams secure the original database, then copy its content into a search index where access rules are coarser, stale or absent. The model can now reveal a document that the application itself would have refused to show.

Every chunk in an index needs enough security metadata to enforce the source system's boundary: tenant, classification, owner and any relevant group or record-level rules. Filter before retrieval, not after generation. Asking the model to omit confidential material after it has received that material is already too late.

Multi-tenant systems should never rely on tenant names in prompts. Scope queries using trusted tenant identifiers derived from the authenticated session, and make cross-tenant retrieval structurally difficult. If permissions can change, decide how quickly the index and its access metadata must reflect revocation. Search freshness is a security property as well as a relevance property.

Embeddings are not harmless metadata. They can reveal relationships and may remain personal or commercially sensitive data. Apply retention, deletion, residency and backup rules to indexes, caches and evaluation datasets-not only to the primary database.

Prompt injection is a containment problem

Prompt injection is often approached as a wording contest: improve the system prompt until malicious instructions stop working. That defence cannot be relied upon. An email, PDF, support ticket or web page may contain instructions that compete with the application's instructions when the content is passed to a model. The attacker does not need direct access to the chat box.

Detection and content filtering reduce risk, but the durable control is limiting what a compromised interaction can accomplish. Give each tool the narrowest function and permissions possible. Prefer GetInvoiceStatus(invoiceId) over a general SQL tool, and CreateRefundProposal over IssueRefund. Validate every argument server-side. Apply rate and value limits. Require explicit approval for external communication, deletion, payment or disclosure of sensitive information.

The same principle applies to agents. More autonomy creates a larger blast radius, not merely a better user experience. If a model can choose tools repeatedly, read untrusted content and act using a powerful service identity, one poisoned document can become an action chain. Limit iterations, destinations, data volume and elapsed time, and make cancellation effective between tool calls.

Human review only works when people can genuinely disagree

Putting a person in the workflow does not automatically make it safe. Reviewers tend to accept polished suggestions, especially when queues are long and the system is usually right. A confirmation button can become ceremonial automation.

Present the source evidence beside the proposal, highlight uncertainty and make edits or rejection as easy as acceptance. Do not use a model's self-reported confidence as though it were a calibrated probability. Confidence thresholds should come from measured performance on representative cases, including the awkward tail that creates most operational damage.

Route reviews according to consequence, not just model score. A low-value categorisation may be safe to automate at a lower confidence than a rare high-value payment. Randomly sample apparently successful cases too; reviewing only low-confidence outputs leaves systematic, confident errors invisible.

Capture the reason for a correction. Those labels reveal whether the problem is missing context, ambiguous policy, poor retrieval, a model limitation or an unsuitable task. A growing review queue is also a product signal: the feature may have moved labour rather than removed it.

Evaluation must version the whole system

A few impressive examples are not an evaluation. Build a test set from real, suitably governed work: ordinary cases, boundary cases, adversarial inputs and known historical failures. Define task-specific measures before comparing models. A document extractor might need field-level precision and recall; a summariser may need factual consistency, required-detail coverage and citation correctness.

The deployable unit is not just the model name. Behaviour depends on the model version, system prompt, tool descriptions, retrieval logic, chunking, filters, temperature, application code and reference data. Record that configuration with each evaluation and production trace. Otherwise a regression cannot be reproduced.

Automated model-based graders are useful for scale, but they can inherit bias and miss business-specific errors. Calibrate them against human judgements and retain deterministic checks wherever possible. Dates, totals, identifiers and required fields should be checked by code rather than awarded a subjective quality score.

Run the evaluation suite when any component changes. Providers can retire or update models, and a seemingly harmless prompt edit can improve average quality while breaking a critical minority case. AI releases need regression gates just as conventional software does; the assertions are simply statistical as well as deterministic.

Privacy starts by sending less

Encryption and contractual assurances matter, but data minimisation is the strongest first control. Remove fields that are irrelevant to the task, replace identifiers where practical and avoid sending a complete case record when three attributes will do. Relevant context improves quality; indiscriminate context increases cost, latency and exposure.

Map every secondary copy: prompts, responses, traces, safety-filter logs, search indexes, caches, support exports and evaluation datasets. Redacting the application log while an observability platform records full model payloads is a common failure. Decide retention and access for each store and verify deletion rather than assuming it propagates.

Confirm how the chosen service handles submitted data, abuse monitoring, regional processing and model training. Those facts are service- and contract-specific and can change; they should be recorded during procurement rather than inferred from the public consumer product. Use managed workload identities where supported instead of distributing long-lived API keys.

Design for graceful degradation, not magical availability

Model endpoints introduce rate limits, regional capacity constraints, latency variation and cost ceilings. The existing application should remain usable when the AI feature is slow or unavailable. For optional assistance, disable the feature cleanly and preserve the normal workflow. For document processing, queue work, expose status and make retries idempotent.

Timeouts require care because abandoning the HTTP request does not necessarily stop provider-side processing or charging. Retries can duplicate cost and actions. Assign an operation identifier, distinguish transient transport failure from rejected content, and never retry a consequential tool call unless its idempotency is proven.

Fallback models are not interchangeable drop-in components. They may interpret prompts, tools and structured schemas differently. Evaluate each fallback and make the user experience honest about reduced capability. Sometimes the safest fallback is no AI result at all.

Observe quality, security and economics together

Traditional availability dashboards cannot tell whether an AI feature is useful. Track task completion, acceptance and correction rates, escalation volume, retrieval failures and sampled factual errors alongside latency and endpoint failures. Segment results by workflow, document type, language and tenant cohort; an acceptable average can conceal a badly served group.

Record model and prompt versions, token usage, retrieved-document identifiers, tool calls, validation outcomes and safety-control decisions. Avoid recording raw sensitive content by default. A trace should let an authorised investigator reconstruct why the system behaved as it did without turning telemetry into a shadow database.

Cost is part of reliability. Put request, token and concurrency budgets around users and tenants, cap context and output sizes, and alert on sudden changes in consumption. A loop that repeatedly calls a model is both a financial incident and a possible security incident.

A rollout pattern that earns autonomy

Progression should be earned by evidence, not scheduled by optimism. Keep an independent kill switch that prevents tool execution or AI traffic without requiring a deployment. Rollback must cover prompts, retrieval configuration and model versions as well as application binaries.

The production-readiness questions

• What is the harm from a false positive, false negative or fabricated answer?
• Which decisions remain deterministic, and which authority has been delegated to the model?
• Can retrieved data ever cross a user, role or tenant boundary?
• How are model outputs schema-validated and semantically validated?
• What happens when untrusted content contains instructions?
• Can a reviewer see the evidence and comfortably reject the proposal?
• Which versioned evaluation proves this exact configuration is releasable?
• Where do prompts, responses, embeddings and traces persist, and for how long?
• How does the core workflow behave during timeout, throttling or provider outage?
• Who can disable the feature, and can they do so without deploying?

If these questions have vague answers, adding a more capable model will not make the feature ready. The hard work is building a trustworthy system around an inherently fallible component.

Related C3 Software guides

• Prompt Injection: What Business Software Teams Need to Know
• Controlling the Cost of AI Features in SaaS Products
• Using AI for Document Processing Without Losing Human Oversight
• Evaluating AI Output Quality in Production Software