The safest way to deploy AI is to assume it will eventually fail in a way that looks convincing. Generative models can hallucinate, expose sensitive information, follow malicious instructions, misuse tools, consume unexpected resources and change behaviour after model or data updates. That does not make AI unusable. It means the design has to expect failure instead of treating failure as an exceptional event. NIST’s AI Risk Management Framework and the OWASP GenAI Security Project both emphasise risk management, monitoring and controls rather than blind trust in model output. A useful practical method is a pre-mortem: imagine the AI project has already caused a serious problem, then ask what had to go wrong for that outcome to happen. Five recurring failure modes cover a large proportion of real-world AI risk.
Failure one: the answer sounds right and is wrong
The most familiar AI weakness is hallucination, but the operational problem is broader than invented facts. A model can misread a document, merge two similar customers, calculate from the wrong period, cite a source that does not support the claim or give a plausible answer to a question it should have refused.
The danger comes from fluency. Humans naturally give more weight to confident, well-structured language, so a polished mistake can pass review more easily than a messy one.
The best control is not a stronger prompt. It is an evidence path. If the answer matters, the workflow should make the underlying source visible. A support summary should link to the ticket history. A financial explanation should point to the actual figures. A RAG system should provide citations that can be inspected.
High-risk outputs should also have deterministic checks where possible. Numbers can be validated against source systems. Product codes can be checked against databases. Policy answers can be constrained to approved documents.
Another practical technique is confidence through abstention. The system should be allowed to say that it lacks enough evidence. A model forced to answer every question will eventually invent.
Teams should also test adversarially. Ask the system questions where the correct answer is absent from the knowledge base and see whether it admits uncertainty.
The rule is simple: use AI for interpretation, but preserve a route back to evidence.
Failure two: the model sees data it should never receive
AI systems often become data pipelines accidentally. Staff paste customer records, contracts, source code, credentials, health information and internal discussions into tools because the interface feels like a private chat window.
That is a governance problem before it is a technical one.
Businesses need clear rules about which systems are approved, what data classifications may be processed and whether information is retained or used for model improvement. Sensitive data should be minimised before it reaches the model.
Retrieval systems create another risk. A user may have permission to ask the chatbot a question but not permission to see every document indexed behind it. Retrieval therefore has to enforce the same access controls as the source systems.
OWASP identifies sensitive information disclosure as a major GenAI risk. The model can leak data directly, infer it from context or repeat information embedded in retrieved documents.
Practical controls include data redaction, tenant isolation, role-based retrieval, secrets management and logging. Prompts should never be treated as a secret-storage mechanism. System prompts can leak too.
The weakest design is a powerful AI connected to a broad data store with no user-level permissions. The strongest design gives the system only the minimum context required for the current task.
Failure three: an agent is allowed to do too much
Agents raise the stakes because they can act, not merely answer.
An agent with email, CRM, filesystem, payment and administrative tools can be extremely useful. It can also create a chain of mistakes faster than a person can notice.
OWASP calls this excessive agency. The risk appears when the agent has more permissions, more autonomy or more tools than the business process requires.
Least privilege should apply to AI exactly as it applies to human accounts. If an agent only needs to draft an email, it should not automatically have send permission. If it needs to read a customer record, it should not have deletion rights.
Separate preparation from execution. The agent can prepare a refund, access change or production deployment, but a human can approve the final action.
Also set boundaries on repetition. An agent that fails should not retry indefinitely, send multiple messages or trigger the same payment twice. Idempotency controls and transaction checks matter.
For tool-using systems, every action should be logged in a way humans can understand. A business cannot govern an agent if nobody can reconstruct what it did.
Autonomy should therefore be earned gradually. Start with read-only, then draft-only, then limited actions, and only widen permissions after measurable reliability.
Failure four: prompt injection turns trusted context against you
Prompt injection occurs when untrusted content changes the behaviour of an AI system. It becomes especially dangerous when an agent can browse websites, read email, retrieve documents or use tools.
A malicious instruction can be hidden in a web page, document or message. The model may interpret that instruction as part of the task instead of as data.
This breaks a common assumption: that retrieved content is trustworthy because it came from a business source. An internal document can still contain untrusted text. A customer email is certainly untrusted.
OWASP continues to rank prompt injection among the most important LLM application risks.
The defence is layered. Treat retrieved text as data, not instruction. Keep tool permissions narrow. Validate outputs before passing them into downstream systems. Use allowlists for actions and destinations. Require confirmation before consequential operations.
Developers should also test indirect injection deliberately. Put malicious instructions inside a document and see whether the agent follows them. Security testing has to include the model’s interpretation layer, not only the application code.
No prompt can guarantee immunity. The system architecture has to make successful injection less damaging.
Failure five: cost, drift and silent degradation arrive after launch
Some AI projects fail slowly rather than dramatically.
Usage rises, prompts become longer, agent loops multiply and monthly cost increases. A model provider changes behaviour. The knowledge base becomes stale. Retrieval quality falls as more documents are indexed. Nobody notices because the system still produces fluent text.
OWASP describes unbounded consumption as a risk, while NIST emphasises ongoing measurement and monitoring. Both ideas matter in ordinary operations.
Set budgets and hard limits. Monitor token consumption, tool calls, latency and error rates. Put ceilings on the number of agent steps. Alert when costs or failure rates move outside normal ranges.
Quality also needs recurring evaluation. Keep a test set of representative questions and rerun it after model, prompt or data changes. Compare answers over time.
Version important prompts and retrieval configurations. If performance drops, the team needs to know what changed.
AI systems should have an owner, not merely a builder. Someone must be responsible for reviewing failures, costs, access permissions and source freshness after launch.
The practical lesson is that weaknesses are manageable when they are expected. Hallucination needs evidence. Data exposure needs minimisation and access control. Agency needs least privilege. Prompt injection needs architecture, not faith in prompts. Cost and drift need monitoring.
AI becomes safer not when it stops failing, but when failure is contained, visible and recoverable.
Sources reviewed September 2026: NIST AI Risk Management Framework and Generative AI Profile; OWASP GenAI Security Project and current LLM/agentic risk guidance.