LLM security testing is a hands-on adversarial assessment of an application built on a large language model. Testers try to make the system leak data, ignore its instructions, misuse its tools or overrun its budget, then report what worked and how to fix it. Most programs map findings to the OWASP Top 10 for LLM Applications (2026).
If your company is about to ship a chatbot, an internal copilot, a retrieval-augmented generation (RAG) search tool or an AI agent, this guide covers what LLM security testing checks and how an engagement runs. It also covers what a good report contains and how to prepare for scoping.
What is LLM security testing?
LLM security testing (also called LLM penetration testing or LLM red teaming) evaluates the whole application around a model, not just the model. That includes the system prompt, the data the model can retrieve, the tools and APIs it can call, the way its output is displayed or executed, and the controls that limit cost and access.
The goal is the same as any penetration test: find exploitable weaknesses before an attacker does. What changes is the attack surface. In an LLM application, plain language is the input, and text the model reads from a document, a web page or an email can act as an instruction.
How is LLM security testing different from a traditional penetration test?
A traditional web application penetration test still matters for an AI feature, because the APIs, authentication and hosting around it can fail in familiar ways. But a traditional test does not check how the model behaves when someone tries to manipulate it.
| Traditional web app penetration test | LLM security testing | |
|---|---|---|
| Main target | Code paths, APIs, authentication, access control | Model behaviour, prompts, retrieval, tools and output handling |
| Attacker input | Crafted requests and payloads | Natural language, documents, images and other content the model reads |
| Typical findings | Injection, broken access control, misconfiguration | Prompt injection, data leakage through answers, excessive agency, unsafe output |
| Repeatability | The same request usually gives the same result | Results vary between attempts, so testers repeat attacks and report how reliably they succeed |
| Common frameworks | OWASP Top 10, OWASP API Security Top 10 | OWASP Top 10 for LLM Applications 2026, OWASP Top 10 for Agentic Applications, MITRE ATLAS |
Most teams shipping an AI feature need both: a conventional test of the application and infrastructure, and an LLM-specific test of the AI layer.
What are the top LLM security risks to test in 2026?
The OWASP GenAI Security Project published the 2026 edition of the Top 10 for LLM Applications on August 4, 2026. It is the most widely used checklist for LLM security testing. The 2026 edition moved Excessive Agency from sixth to third and renamed System Prompt Leakage to Hidden Context Exposure.
| OWASP 2026 ID | Risk | What testers try to do |
|---|---|---|
| LLM01 | Prompt Injection | Override the model’s instructions directly, or hide instructions in documents, web pages, images or emails the model reads |
| LLM02 | Sensitive Information Disclosure | Get the model to reveal personal data, credentials or records the user is not allowed to see |
| LLM03 | Excessive Agency | Make the model call tools, send messages, change data or hit internal APIs beyond what its task needs |
| LLM04 | Supply Chain | Identify risky third-party models, plugins, libraries and connectors in the stack |
| LLM05 | Data and Model Poisoning | Check whether training, fine-tuning or knowledge-base data could be tampered with to change behaviour |
| LLM06 | Unbounded Consumption | Trigger runaway token use, long tool chains or costly requests that drive up spend or deny service |
| LLM07 | Misinformation | Push the model into confident, wrong answers that users or downstream systems act on |
| LLM08 | Hidden Context Exposure | Extract the system prompt, tool definitions, configuration or other hidden context |
| LLM09 | Vector and Embedding Weaknesses | Read another tenant’s data through retrieval, or plant content in the vector store |
| LLM10 | Improper Output Handling | Get model output to run as code, query or markup in a browser, shell, database or terminal |
If your feature uses agents with tools and memory, OWASP’s separate Top 10 for Agentic Applications covers those risks in more depth.
What does an LLM security testing engagement look like?
Most engagements follow three stages. At Canary Trap they are called Define, Uncover and Report.
- Define. Agree on scope, objectives and rules of engagement. This covers which applications, models, user roles, tools and data sources are in scope, which environment is tested, and how test data is handled. A short threat model identifies what an attacker would want: data, actions, money or reputation.
- Uncover. Testers attack the application the way a real adversary would. They combine manual, creative testing with automated adversarial prompt sets. They work through each in-scope risk, chain weaknesses together, and repeat successful attacks to see how reliably they work.
- Report. Findings are written up with evidence, business impact, severity and specific remediation steps, and mapped to the OWASP 2026 categories.
Testing can run against a staging environment or production. The choice depends on how closely staging mirrors production data, tools and permissions, and it should be settled in the rules of engagement.
How do you test a RAG application for prompt injection and data leakage?
RAG applications answer questions using your own documents, so the main risks are about what goes into the knowledge base and who can get it back out. A RAG security test typically checks:
- Indirect prompt injection: whether a document, ticket or web page in the knowledge base can carry hidden instructions the model follows.
- Authorization before retrieval: whether a user can get answers drawn from documents they don’t have permission to open.
- Tenant and session isolation: whether one customer’s or one user’s data can surface in another’s answers.
- Poisoned content: whether someone with write access to a source system can plant content that changes answers for everyone.
- Data exfiltration paths: whether secrets in prompts, logs, environment variables or connected systems can be pulled out through responses, links or rendered images.
The OWASP RAG Security Cheat Sheet is a useful companion for engineering teams fixing these issues.
How do you test AI agents for unsafe tool use?
An AI agent that can send emails, run scripts, create tickets or call internal APIs turns a bad answer into a bad action. Agent testing focuses on what the model can do, not just what it says:
- Can a prompt, or content the agent reads, make it call a tool it shouldn’t?
- Does each tool run with the narrowest permissions and shortest-lived credentials the task needs?
- Do irreversible actions (deleting data, sending money, merging code, emailing customers) require human approval that shows the real action?
- Can an attacker trigger loops or long tool chains that run up cost?
- Do connectors, plugins or Model Context Protocol (MCP) servers expose more access than intended?
Excessive Agency climbed to third place in the OWASP 2026 list, which reflects how much risk now sits in what AI systems are allowed to do.
Automated LLM scanning vs human red teaming: do you need both?
Yes, for most production AI features. They answer different questions.
| Automated LLM scanning | Human-led LLM red teaming | |
|---|---|---|
| Best at | Running large sets of known attack prompts quickly and repeatedly | Context-specific attacks, chained exploits and business-logic abuse |
| Typical tools | Open-source frameworks such as NVIDIA garak and promptfoo, plus commercial scanners | Experienced testers using those tools plus their own techniques |
| When to use | In CI/CD on every model, prompt or tool change | Before launch, after major changes and on a regular schedule |
| Limitation | Finds what it was built to find; misses attacks specific to your app | Point-in-time unless repeated |
A practical setup runs automated adversarial tests in the pipeline to catch regressions, and brings in a human-led test before launch and after significant changes.
What should an LLM security testing report include?
A good report lets engineers fix issues and lets leaders understand the risk. Ask for:
- An executive summary in plain language, with overall risk and the most important fixes
- The exact scope tested, including the model, provider, version, system prompt version, tools and data sources
- Each finding mapped to an OWASP Top 10 for LLM Applications 2026 category
- Reproduction evidence: the prompts, documents or inputs used and the responses they produced
- How reliably each attack worked across repeated attempts
- Business impact and a severity rating for each finding
- Specific remediation guidance, such as permission changes, retrieval controls, output encoding or approval steps
- What was not tested, so gaps are visible
Does LLM security testing help with SOC 2, ISO 27001 and AI regulations?
Yes, as evidence. Frameworks such as SOC 2, ISO 27001 and ISO/IEC 42001 don’t prescribe a specific LLM test, and neither does the NIST AI Risk Management Framework. But each expects you to identify risks to your systems and show that controls work. A documented LLM security test gives you that evidence for auditors and for customer security reviews.
The NIST AI 600-1 Generative AI Profile and the EU AI Act both point toward adversarial testing for generative AI. The OWASP 2026 list now includes a single appendix mapping each risk to external frameworks, including NIST and MITRE ATLAS. That makes it easier to connect test findings to your compliance controls. For how penetration testing supports SOC 2 more broadly, see our guide to SOC 2 penetration testing.
How is LLM security testing scoped?
Scope, not the model, drives the effort. A single customer-support chatbot with no tools is a very different test from a multi-tenant RAG platform with agents that act on internal systems. The main factors are:
- Number of AI applications or features in scope
- Architecture: chatbot, RAG, agent, or a mix
- Number of tools, plugins, connectors and data sources
- Number of user roles and tenants
- Input types: text only, or also voice, images and files
- Environment: staging, production or both
To get an accurate proposal quickly, have these ready: an architecture diagram, the model provider and version, the system prompts, a list of tools with their permissions, the data sources, test accounts for each user role, and any compliance driver such as SOC 2 or a customer security review. Canary Trap’s scoping questionnaire walks through each of these.
When should you test an LLM application?
Test before launch, then again whenever the risk changes:
- Before an AI feature goes live to customers or employees
- When you switch model, provider or model version
- When you add a new tool, connector, plugin or data source
- After a major change to system prompts or guardrails
- At least once a year for production AI features
- After a security incident involving an AI system
Between tests, automated adversarial checks in CI/CD catch regressions early.
Get your AI application tested
Canary Trap’s AI penetration testing assesses LLM applications, RAG pipelines and AI agents against the OWASP Top 10 for LLM Applications 2026, and delivers a report your engineers and auditors can use. Complete our scoping questionnaire or contact our team to plan your test.
Frequently asked questions about LLM security testing
What is the difference between LLM red teaming and LLM penetration testing?
The terms are often used interchangeably. LLM penetration testing usually means a scoped assessment that works through known risk categories such as the OWASP Top 10 for LLM Applications. LLM red teaming often describes a goal-based exercise, such as “extract customer data” or “make the agent send an unauthorized email,” using whatever path works.
Do I need LLM security testing if we use a third-party model like OpenAI, Anthropic or Google?
Yes. The model provider secures its model, but prompt injection, data leakage, excessive agency, cost abuse and unsafe output handling are properties of your application. They depend on your prompts, data, tools and permissions, which only you control.
What is the OWASP Top 10 for LLM Applications 2026?
It is a free, community-built list of the ten most critical security risks in applications built on large language models, published by the OWASP GenAI Security Project on August 4, 2026. It runs from LLM01 Prompt Injection to LLM10 Improper Output Handling and is the most common checklist for LLM security testing.
Can LLM security testing run in a CI/CD pipeline?
Automated adversarial tests can. Tools such as promptfoo and NVIDIA garak can run attack prompt sets on every change to a model, prompt or tool and fail the build on regressions. They complement, rather than replace, human-led testing before launch and after major changes.
How often should LLM applications be security tested?
Before launch, then whenever the model, provider, tools, data sources or system prompts change significantly, and at least once a year for production features. Automated checks in CI/CD cover the gaps between full tests.
Does an LLM security test help with SOC 2?
Yes. SOC 2 doesn’t name a specific AI test, but it expects you to identify risks and show that controls work. A documented LLM security test with findings and remediation gives auditors and customer security reviewers that evidence for your AI features.
What information do I need to get an LLM security testing quote?
An architecture overview, the model provider and version, the number of AI features in scope, the tools and data sources each one uses, the user roles involved, the environment to be tested, and any compliance deadline. Canary Trap’s scoping questionnaire collects these in one place.
