The model is only one part of the system
AI security discussions often collapse into a narrow question: “Can the model be tricked?” That matters, but it is only one layer of the problem.
A production AI system may retrieve internal documents, call external APIs, access databases, trigger workflows, send email, create tickets, generate code or make recommendations that humans act on. Once those capabilities exist, the security boundary extends far beyond the model itself.
The assessment therefore needs to follow the full path from user input to model behaviour to tool execution to data access to downstream business impact.
Prompt injection matters most when the model has power
Prompt injection is not automatically catastrophic. Its impact depends on what the model can access and what it can do.
If an internal assistant can only answer questions from a public knowledge base, a successful manipulation may have limited consequence. If the same assistant can read sensitive documents, call business APIs or operate with privileged service credentials, the same class of attack becomes much more serious.
This is why testing must evaluate capability boundaries. The question is not simply whether the model follows a malicious instruction. The question is whether that behaviour can cross a trust boundary, expose data, invoke a sensitive tool or influence a high-risk decision.
The more authority an AI system has, the more important it becomes to validate the controls around that authority.
Data exposure is often more realistic than model compromise
Many AI systems are designed to retrieve and transform information. That makes data handling one of the most important areas to test.
An assessment should examine whether the system can reveal documents outside the user’s authorisation scope, whether retrieval filters enforce access control correctly, whether embeddings or indexes include sensitive material, whether logs retain prompts containing confidential data and whether generated responses can leak information from previous sessions or other users.
Testing also needs to consider data ingestion. If users, vendors or automated feeds can add content to a knowledge base, that content may influence later model behaviour. Poisoned or malicious content can therefore become an indirect instruction channel.
Agents create a new class of operational risk
Agents are different from chatbots because they can act.
An agent that can search the web, query internal systems, call an API, execute code or modify records creates a chain of delegated authority. Each tool call needs its own authentication, authorisation, validation and auditability.
Assessors should test whether the agent can call tools outside the intended sequence, supply unexpected parameters, abuse one tool to influence another, execute actions without sufficient user confirmation or operate with broader permissions than the initiating user.
They should also test failure modes. What happens when a tool returns malformed data? What if an external source contains malicious instructions? What if the model becomes uncertain but the workflow still allows an irreversible action?
The cloud and application layer still matter
An AI application is still an application.
Traditional weaknesses remain relevant: insecure APIs, broken authorisation, exposed secrets, vulnerable dependencies, weak session management, insecure storage, excessive cloud permissions and missing monitoring.
In fact, AI features can make ordinary weaknesses more dangerous by connecting systems that were previously separate. A poorly protected API key may now give an agent access to a business system. An overprivileged service account may let a retrieval component read an entire document repository.
A credible AI security assessment therefore combines AI-specific testing with application, API, cloud and identity security review.
Governance should be tested against the real implementation
AI governance documents often describe intended controls: human oversight, acceptable use, data restrictions, approval requirements and model monitoring. The assessment should verify whether the implementation actually enforces those expectations.
If policy says sensitive data must not be submitted to an external model, can users still do it? If high-risk actions require human approval, can the workflow be manipulated to bypass that step? If certain datasets are restricted, does retrieval enforce the same access model?
The gap between policy and implementation is where governance becomes a security issue rather than a documentation issue.
What the final assessment should deliver
The output should not be a list of exotic AI attack names. It should explain which attack paths are realistic, which controls prevented abuse, which capabilities create the most risk and what should change before the system receives more access or wider deployment.
For leadership, the useful question is whether the AI system can be trusted with the authority it currently has. For technical teams, the useful answer is a set of concrete controls around identity, data, tools, prompts, application architecture and monitoring.
That is the difference between testing an AI feature and assessing an AI system.
