Prompt injection testing is how security teams find out whether LLM-powered applications can be manipulated before attackers do. As organizations add AI features to customer portals, internal tools, support workflows, and agentic applications, the model becomes part of the attack surface.
Traditional application security testing still matters. Authentication, authorization, APIs, session management, input validation, and infrastructure exposure all still need to be assessed. But LLM-powered applications introduce another layer of risk: the model can be influenced through natural language, retrieved content, tool outputs, and user-supplied instructions.
The issue isn’t that the model is “bad.” It’s that the model is designed to follow instructions — and attackers can exploit that behavior.
Key Takeaway: Prompt injection isn’t a single vulnerability. It’s a class of attack that appears in predictable patterns when LLM-powered applications retrieve content, call tools, process user input, or expose model-generated output.
What Is Prompt Injection?
Prompt injection is a technique where an attacker manipulates an LLM’s behavior through crafted input. That input may come directly from a user or indirectly through content the model retrieves, summarizes, analyzes, or acts on.
OWASP maintains an actively updated resource on prompt injection and LLM attack patterns that security teams can reference alongside their testing methodology.
Prompt injection may attempt to:
- Override system instructions
- Bypass guardrails
- Reveal sensitive prompts or data
- Influence tool calls
- Manipulate output
- Trigger unauthorized actions
- Cross data boundaries between users, tenants, or document sets
In a traditional application, malicious input targets a parser, database, or API. In an LLM-powered application, malicious input can also target the model’s instruction-following behavior. Standard AppSec testing doesn’t cover that layer.
The Three Production Patterns
| Pattern | How It Works | Primary Risk |
|---|---|---|
| Indirect prompt injection | Malicious instructions embedded in retrieved content | Model follows attacker instructions without user knowledge |
| Tool-call hijacking | User input causes model to call tools with unauthorized arguments | Unauthorized data access, unintended workflow execution |
| System prompt leakage | Hidden instructions exposed through errors or model output | Attackers learn how to bypass or manipulate the system prompt |
Pattern 1: Indirect Prompt Injection Through Retrieved Content
Many LLM-powered applications retrieve external or internal content before producing an answer — from uploaded documents, knowledge bases, support tickets, web pages, emails, chat transcripts, customer records, internal wikis, and RAG pipelines.
Indirect prompt injection happens when retrieved content contains instructions aimed at the model rather than the user. A document may include hidden or visible text telling the model to ignore previous instructions, reveal sensitive information, change its output, or take an unintended action. The user never sees the malicious instruction. The model does.
That creates risk when retrieved content is treated as trusted context.
How to reduce the risk
Retrieved content should be treated as untrusted input, not trusted context. Security teams should evaluate whether the application:
- Separates system instructions from retrieved content
- Prevents retrieved content from influencing authorization decisions
- Limits whether retrieved content can trigger tool calls
- Sanitizes or constrains model output
- Enforces data boundaries outside the model
- Logs the retrieved source material used in responses
- Prevents cross-tenant or cross-user document leakage
The model should be allowed to use retrieved content as information. It shouldn’t allow retrieved content to become instruction authority.
Pattern 2: Tool-Call Hijacking Through User-Supplied Arguments
LLM-powered applications become more capable when models can call tools — searching files, querying databases, creating tickets, sending messages, updating records, triggering workflows, calling APIs. They also become riskier.
Tool-call hijacking occurs when user input causes the model to call a tool with unintended arguments: a different user ID, a different account, a different file path, an unauthorized workflow action, or parameters outside the intended allow-list.
The problem isn’t only that the model called the tool. It’s that the tool may trust the model-generated arguments.
How to reduce the risk
Tool arguments generated by the model should be treated as untrusted user input. Security teams should evaluate whether the application:
- Validates tool arguments server-side
- Schema-checks every field
- Rejects unexpected parameters
- Uses allow-lists for actions and resources
- Enforces object ownership at the tool boundary
- Scopes tool calls to the invoking user
- Prevents arbitrary query, file path, or command execution
- Logs tool calls and authorization failures
The model can suggest what should happen. The application should decide what is allowed to happen.
Pattern 3: System Prompt Leakage Through Errors and Outputs
System prompts often contain role instructions, workflow rules, data handling logic, tool descriptions, safety constraints, and business logic. System prompt leakage occurs when those instructions become visible to users — through raw error messages, stack traces, debug responses, logging mistakes, model output that repeats hidden instructions, or poorly handled exception states.
Once attackers understand the system prompt, they can design inputs that bypass or manipulate it more effectively.
A system prompt should never be treated as the only security boundary. But leaking it still makes abuse easier.
How to reduce the risk
Security teams should evaluate whether the application:
- Prevents raw model errors from reaching users
- Sanitizes exception handling
- Avoids exposing hidden instructions through output
- Logs internally without revealing sensitive prompt content
- Uses generic user-facing error responses
- Avoids placing secrets or sensitive logic in prompts
- Treats prompt secrecy as a contributing layer, not a complete defense
Prompt confidentiality isn’t a security strategy. But unnecessary prompt leakage is still unnecessary risk.
Why Prompt Injection Testing Matters Before Launch
Prompt injection testing evaluates how an LLM-powered feature behaves under adversarial input. A useful assessment covers:
- Direct prompt injection
- Indirect prompt injection
- Jailbreak attempts
- Data leakage
- Retrieval poisoning
- Unsafe tool use
- Cross-user or cross-tenant data exposure
- System prompt leakage
- Output handling
- Model behavior across edge cases
- Authorization enforcement outside the model
The goal isn’t to prove that every possible prompt has been blocked — that’s not realistic. The goal is to identify where model behavior, application logic, tool access, and data boundaries create meaningful business risk.
Why Standard Application Testing Misses These Issues
A traditional application penetration test evaluates authentication, authorization, input validation, session management, APIs, and application logic. It may not assess how an LLM behaves when instructions are embedded in retrieved content, tool arguments are influenced by user prompts, or model-generated outputs affect downstream workflows.
An AI-powered feature can pass standard AppSec testing and still be vulnerable to prompt injection. LLM security testing should complement application penetration testing, not replace it.
How Canary Trap Can Help
Canary Trap helps organizations evaluate AI-powered applications, LLM features, and agentic workflows before they create production exposure. Depending on the architecture, this may include:
- AI / LLM Security Testing
- Application Penetration Testing
- API Penetration Testing
- Secure Code Review
- Red Team Exercise
- Purple Team Exercise
These assessments identify whether prompt injection, unsafe tool use, retrieval poisoning, weak authorization, or output handling issues create exploitable risk. The right assessment depends on how the AI feature is built, what data it can access, which tools it can call, and what business workflows it supports.
Prompt Injection Requires Validation, Not Assumption
There is no system prompt that makes an LLM application secure on its own. The model is part of the attack surface — it should not be treated as the security boundary.
Effective defense requires layers: treating retrieval as untrusted, validating tool arguments server-side, scoping permissions tightly, designing tools safely, handling output carefully, maintaining logging, and testing with human-led adversarial techniques. None of those layers are exotic. Together, they make the system meaningfully harder to abuse.
If your organization is preparing to launch an LLM-powered application, chatbot, RAG workflow, or AI agent, Canary Trap can help validate where prompt injection risk appears before attackers do.
Schedule a scoping conversation with Canary Trap to discuss your AI application architecture and LLM security testing objectives.
Frequently Asked Questions
What is prompt injection testing?
Prompt injection testing evaluates whether an LLM-powered application can be manipulated through crafted input, retrieved content, tool calls, or unsafe outputs. It identifies where model behaviour creates security or business risk that standard application testing doesn’t catch.
What is indirect prompt injection?
Indirect prompt injection occurs when malicious instructions are embedded in content the model retrieves or processes — documents, web pages, tickets, emails, or knowledge base entries. The user doesn’t send the malicious input; the model encounters it through its retrieval pipeline.
Can a system prompt prevent prompt injection?
No. A strong system prompt can guide model behaviour but shouldn’t be treated as a security boundary. Server-side validation, scoped permissions, retrieval safeguards, and output handling are still required. Prompt engineering is one layer, not a complete defense.
Why does prompt injection matter for RAG applications?
RAG applications retrieve content from external or internal sources and pass it to the model as context. If that content contains malicious instructions or crosses data boundaries between users or tenants, the model may produce unauthorized, misleading, or unsafe outputs.
What is tool-call hijacking?
Tool-call hijacking occurs when user input influences a model to call a tool, API, or function with unauthorized or unexpected arguments. It’s especially risky when tools can access sensitive data, modify records, or trigger business workflows — and when those tools trust model-generated arguments without server-side validation.
What is retrieval poisoning?
Retrieval poisoning occurs when an attacker introduces malicious content into a knowledge base, document store, or other retrieval source that an LLM-powered application queries. When the model retrieves and acts on that content, it may follow attacker-supplied instructions rather than the system’s intended behaviour.
When should organizations test for prompt injection?
Before launching AI features, after changing prompts or retrieval sources, after adding tools or APIs, and before expanding access to sensitive data or high-risk workflows. For applications in active development, testing should occur at meaningful milestones — not only before initial launch.
How is prompt injection different from traditional injection attacks like SQL injection?
SQL injection targets a database parser by embedding SQL commands in user input. Prompt injection targets an LLM’s instruction-following behaviour by embedding natural language instructions in input or retrieved content. The mechanism differs — one exploits a parser, the other exploits the model’s core design — but the underlying pattern is the same: untrusted input influencing application logic in unintended ways.
Does prompt injection testing require AI-specific expertise?
Yes. Effective prompt injection testing requires understanding how LLMs process context, follow instructions, and interact with tools and retrieval systems. Traditional application penetration testing methodology covers important ground but doesn’t address model-specific attack patterns like indirect injection, jailbreaking, or tool-call manipulation.
