How to Security Test an AI App Before You Ship It

Shipping an AI feature is no longer unusual. Testing one before it ships still is.

Most security teams apply traditional pentest thinking to LLM-powered applications — scope the app, run the standard methodology, sign off. That approach catches the vulnerabilities it knows how to look for. It misses an entire category of risk that doesn’t exist in a standard web application, because that category didn’t exist before the model became part of the attack surface.

Why Standard AppSec Testing Misses AI-Specific Risk

A traditional penetration test is built to find authentication flaws, injection vulnerabilities, broken access control, and API exposure. All of that still matters in an AI-powered app — the model sits on top of infrastructure that still needs to be tested conventionally.

But LLMs introduce risk categories that conventional methodology wasn’t designed to catch:

• Prompt injection — manipulating model behavior through crafted input
• Jailbreaking — bypassing safety constraints built into the model or system prompt
• Data exfiltration through model outputs — getting the model to leak training data, system prompts, or other users’ information
• Unsafe tool use — the model taking real-world actions it shouldn’t, through connected APIs or functions
• Retrieval poisoning in RAG architectures — manipulating the documents a model retrieves from to influence its output

The OWASP LLM Top 10 is a reasonable starting framework for understanding the shape of this risk category, but a framework isn’t a test plan. It tells you what to look for, not how deep to look.

What to Test Before Launch, by Architecture

LLM security risk isn’t uniform — it depends heavily on how the model is deployed.

Chatbot / customer support: Test for prompt injection, jailbreak resistance, and PII leakage in responses. These are the most common deployment pattern and the most commonly under-tested.

RAG app with documents: Test for retrieval poisoning, document-based injection, and whether the model respects data boundaries between users or document sets it shouldn’t cross.

Agentic workflows (tool use, API calls): Test for unsafe function calling, privilege escalation through the LLM itself, and lateral movement via connected systems. This is the highest-risk category and the least frequently tested.

Human Red Teaming vs. Automated Scanning

Automated scanning finds known attack patterns quickly and cheaply. It’s useful as a baseline and a continuous check between releases.

Human red teaming finds the novel attack paths and business logic issues automated tools aren’t built to recognize — the chained prompt that gets past a filter, the tool-calling sequence that wasn’t supposed to be possible.

For production applications with real users, automated scanning alone isn’t sufficient. The question isn’t which one to use — it’s when to use each, and most mature programs use both: automated scanning continuously, human-led testing at launch and after significant changes.

What a Good LLM Security Testing Engagement Includes

• Scope definition covering the model, system prompt, tooling, and integrations — not just the application shell
• Test case categories spanning prompt injection, jailbreaks, data exfiltration, tool abuse, and model extraction
• Deliverables mapped to the OWASP LLM Top 10, with risk-prioritized remediation and defined retest scope
• A realistic timeline for a single-app assessment — ask your vendor directly what this looks like for your architecture

If a proposed engagement doesn’t name these elements explicitly, ask why. “We’ll test your AI feature” is not a scope.

Conclusion

Organizations shipping AI features face a security testing gap that most vendors aren’t built to close. Most test the application layer. Few test the model interaction layer where prompt injection, jailbreaks, and tool abuse actually live.

Before you ship, know which layer your provider actually covers — and make sure the report you get back is specific enough to act on, not just a checkbox confirming “AI testing was performed.”

Use this as a check against your current process . Talk to our team about what an LLM security assessment looks like for your architecture.

---

Frequently Asked Questions

What is LLM security testing?

LLM security testing evaluates how an AI-powered application behaves when the model becomes part of the attack surface. It looks beyond standard application vulnerabilities to assess risks such as prompt injection, jailbreaks, data leakage, unsafe tool use, retrieval poisoning, and unintended model behaviour.

Is LLM security testing different from a traditional penetration test?

Yes. Traditional penetration testing focuses on the application, infrastructure, authentication, access control, APIs, and other conventional attack paths. Those still matter. LLM security testing adds another layer by assessing how the model, prompts, retrieval systems, connected tools, and user interactions can be manipulated or abused.

Why isn’t standard AppSec testing enough for AI-powered applications?

Standard AppSec testing is not designed to fully assess model-specific risks. It may identify issues in the application shell, but it can miss vulnerabilities that happen through natural language interaction, model behaviour, retrieval workflows, or tool/function calls. An AI feature can pass a conventional test and still be vulnerable to prompt injection, data leakage, or unsafe automation.

What should be tested before launching an LLM-powered feature?

The scope should include the application layer, the model interaction layer, the system prompt, retrieval sources, connected tools or APIs, data boundaries, and the ways users can influence model behaviour. The exact test plan should depend on the architecture, whether the feature is a chatbot, RAG application, agentic workflow, internal copilot, or customer-facing AI assistant.

What is prompt injection?

Prompt injection is a technique where a user attempts to manipulate the model’s behaviour through crafted input. This can include trying to override instructions, bypass restrictions, expose hidden prompts, retrieve sensitive information, or cause the model to take an unintended action.

What is retrieval poisoning in a RAG application?

Retrieval poisoning happens when the documents or data sources used by a retrieval-augmented generation system are manipulated to influence the model’s output. If the model retrieves from compromised, untrusted, or poorly segmented sources, it may produce misleading, unauthorized, or unsafe responses.

Why are agentic AI workflows higher risk?

Agentic workflows are higher risk because the model may be connected to tools, APIs, databases, ticketing systems, code repositories, or other business systems. If the model can take action, not just generate text, testing needs to evaluate whether it can be manipulated into doing something it should not do.

Can automated AI security scanning replace human red teaming?

No. Automated scanning is useful for identifying known patterns and running repeatable checks, especially between releases. Human red teaming is better suited to finding novel attack paths, chained behaviours, business logic issues, and context-specific risks that automated tools may miss. Mature testing programs often use both.

When should an organization test an AI feature?

Organizations should test AI features before launch, after major changes to prompts, models, retrieval sources, tools, permissions, or integrations, and before expanding access to more users or higher-risk workflows. Testing is especially important when the AI feature interacts with sensitive data or can trigger actions in other systems.

What should a good LLM security testing report include?

A useful report should identify the tested architecture, attack paths, findings mapped to relevant frameworks such as the OWASP LLM Top 10, business impact, technical evidence, remediation guidance, and retest scope. It should make clear what was tested, what was exploitable, what matters most, and what needs to change before launch.

Share post: