AI Security · LLM · Adversarial ML

AI Security Research

OWASP Top 10 for LLMs, adversarial machine learning, prompt injection taxonomy and defensive AI techniques. Reference for practitioners securing AI-powered applications.

OWASP LLM Top 10 ↗·MITRE ATLAS ↗·AI Village ↗·Knowledge Base: Prompt Injection Deep Dive →

OWASP Top 10 for Large Language Models (2025)

LLM01

Prompt Injection

Critical▾

Attacker input alters the LLM's behaviour or output in unintended ways — RAG and fine-tuning reduce but do not eliminate the risk, and inputs need not be human-readable, only model-parseable.

Attack Types

·Direct injection: a user's own input overrides the system prompt
·Indirect injection: instructions hidden in a retrieved document, webpage or file the model summarises
·Multimodal injection: instructions hidden in an image or other non-text input processed alongside benign text

Mitigations

·Dual-LLM pattern: a privileged orchestrator never sees untrusted content directly, and delegates parsing to a quarantined LLM with no tool access
·Typed tool calls (e.g. via MCP) with schema-validated arguments, so a model can't smuggle instructions through a free-text parameter
·Require human approval before any high-risk or irreversible action
·Segregate and clearly mark untrusted external content before it reaches the model
·Give the application its own scoped API tokens rather than letting the model hold broad credentials
·Treat this as risk reduction, not elimination — UK NCSC's Dec 2025 assessment concluded prompt injection may never be fully mitigated in current LLM architectures

LLM02

Sensitive Information Disclosure

High▾

LLMs and the applications around them can expose PII, proprietary algorithms, or confidential business data through memorised training data, system prompts, or retrieval context.

Attack Types

·PII or credential leakage memorised from training data
·Proprietary model or algorithm exposure via inversion attacks (e.g. the 2019 'Proof Pudding' CVE-2019-20634)
·Cross-user context leakage in shared multi-tenant deployments

Mitigations

·Sanitise data before it enters training or fine-tuning
·Enforce least-privilege access controls on data sources the model can reach
·Apply differential privacy to reduce memorisation of individual records
·Give users clear opt-out and data-retention policies

LLM03

Supply Chain

High▾

LLM supply chains extend traditional software risk to third-party pre-trained models, datasets, and fine-tuning adapters — any of which can be tampered with before they reach production.

Attack Types

·Weak model provenance: a compromised or impersonated model-repo account distributes a backdoored model
·Malicious LoRA adapters that compromise the base model they're merged or bolted onto
·Outdated, deprecated, or vulnerable third-party ML components and licensing conflicts

Mitigations

·Verify provenance and hash pre-trained weights before loading them
·Track datasets and models with an ML-BOM (e.g. OWASP CycloneDX)
·Pin dependency versions and run SCA scanning in the ML CI/CD pipeline
·Prefer models and adapters from vetted, actively maintained sources

LLM04

Data and Model Poisoning

High▾

Manipulated pre-training, fine-tuning, or embedding data introduces backdoors, bias, or degraded performance — a poisoned trigger can leave a model behaving normally until it fires, making it hard to test for.

Attack Types

·Split-view or frontrunning poisoning exploiting training-data collection dynamics
·Malicious pickled model files that execute code when loaded
·Backdoor triggers embedded in fine-tuning data that activate only on a specific input pattern

Mitigations

·Track data origin and transformations with data version control or an ML-BOM
·Sandbox and vet unverified external data sources before ingestion
·Monitor training loss for anomalies that indicate poisoning
·Red-team fine-tuned models before deployment

LLM05

Improper Output Handling

High▾

Insufficient validation or sanitisation of LLM output before it's passed downstream — distinct from Overreliance, which is about trusting the content's accuracy rather than its handling — can enable XSS, SSRF, SQL injection, or remote code execution.

Attack Types

·LLM output piped directly into a shell, eval(), or similar sink → remote code execution
·Unescaped LLM-generated HTML/Markdown/JavaScript rendered in a browser → XSS
·LLM-constructed SQL executed without parameterisation → SQL injection

Mitigations

·Treat model output as untrusted input — apply the same zero-trust validation as any external user
·Use context-aware output encoding (HTML, JS, SQL) for wherever the output lands
·Use parameterised queries or prepared statements for any database operation
·Apply a strict Content Security Policy as a backstop against residual XSS

LLM06

Excessive Agency

High▾

An LLM-based agent granted more functionality, permissions, or autonomy than its task requires turns a manipulated or hallucinating model into one that can actually cause damage.

Attack Types

·Excessive functionality: an extension exposes delete/modify capability when only read access was needed
·Excessive permissions: an extension connects downstream with a shared high-privilege identity instead of a per-user scoped one
·Excessive autonomy: a multi-agent or agentic chain executes several steps with no human checkpoint

Mitigations

·Grant each extension only the minimum functionality and permissions its task needs
·Scope downstream credentials per-user rather than through one privileged shared identity
·Require human-in-the-loop confirmation before irreversible actions
·Log every agentic action for audit and review

LLM07

System Prompt Leakage

Medium▾

System prompts can end up holding secrets or internal rules that were never meant to depend on secrecy. The real risk isn't the disclosure itself — it's designing the application to rely on the prompt as a security boundary in the first place.

Attack Types

·Credentials, API keys, or connection strings embedded directly in the prompt text
·Internal business rules disclosed (e.g. a stated transaction limit) that tell an attacker exactly what to bypass
·Content-filtering criteria revealed, handing attackers a map for crafting a bypass

Mitigations

·Never store secrets or access-control logic inside the system prompt
·Enforce authorization and session management in application code, never by delegating it to the model
·Treat the system prompt as inherently non-secret from day one
·Test for prompt extraction during red-team exercises

LLM08

Vector and Embedding Weaknesses

High▾

RAG pipelines introduce risk in how vectors and embeddings are generated, stored, and retrieved — weak isolation or unvalidated sources can leak data across tenants or let an attacker poison what gets retrieved.

Attack Types

·Unauthorised cross-tenant access to a shared vector database exposing another user's documents
·Embedding inversion attacks that reconstruct source text from stored vectors
·Poisoned documents injected into the knowledge base to manipulate retrieval results

Mitigations

·Apply fine-grained, permission-aware access control on the vector store per tenant/user
·Validate and authenticate data sources before they enter the index
·Regularly audit the knowledge base for injected or anomalous content

LLM09

Misinformation

Medium▾

LLMs produce false or misleading output that reads as credible, largely driven by hallucination and compounded by user overreliance — real incidents include Air Canada's chatbot inventing a refund policy the airline was held to, and fabricated case law cited in an actual US court filing.

Attack Types

·Factual inaccuracies stated with unwarranted confidence
·Fabricated citations, case law, or non-existent CVEs presented as real
·Unsafe or non-existent third-party libraries suggested in generated code

Mitigations

·Ground responses in retrieval-augmented generation from vetted sources rather than model memory alone
·Require human review before acting on high-stakes output
·Cross-verify LLM claims against authoritative sources before trusting them

LLM10

Unbounded Consumption

Medium▾

Letting users drive uncontrolled inference enables denial of service, runaway API costs ('denial of wallet'), or model extraction through high-volume querying — the previous Model DoS and Model Theft risks share this one root cause.

Attack Types

·Variable-length input flooding that exploits processing inefficiencies
·Denial-of-wallet attacks exploiting pay-per-use pricing on cloud AI services
·Model extraction via high-volume API querying to train a distilled replica

Mitigations

·Enforce strict input size limits and per-key rate limiting/quotas
·Restrict or obfuscate exposure of logits/logprobs that aid extraction
·Set up cost-based alerting on anomalous per-user API spend

Adversarial Machine Learning in Security Tooling

Attack Class	Target Systems	Technique	Defence
Evasion Attack	Malware classifiers, WAF ML rules, spam filters	Craft input that crosses decision boundary while remaining functional. FGSM, PGD, C&W attacks.	Adversarial training, input denoising, ensemble methods, certified robustness bounds
Poisoning Attack	IDS/anomaly detection models trained on live traffic	Inject malicious traffic that shifts decision boundary during online retraining.	Data provenance validation, anomaly detection on training data, periodic offline retraining
Model Inversion	Security classifiers that expose training data via confidence scores	Iterative queries reconstruct training examples (network configs, user data) from model outputs.	Output confidence score suppression, differential privacy, output space discretisation
Backdoor Attack	Fine-tuned models, federated learning security systems	Embed trigger pattern during training — misclassification fires only when trigger is present.	Neural Cleanse, STRIP defence, activation clustering anomaly detection, dataset verification