AI red teaming and AI security testing glossary
18 terms buyers meet when evaluating AI red teaming vendors.
- AI red teaming
- Deliberate adversarial testing of an AI model, application, or agent to find insecure, unsafe, or unwanted behavior before attackers or users do.
- Prompt injection
- An attack where crafted input makes a model ignore its instructions or follow the attacker’s instructions instead.
- Indirect prompt injection
- Prompt injection delivered through content the system reads, such as a web page, document, email, or tool result, rather than typed by the user.
- Jailbreak
- An input or conversation that gets a model to produce output its safety rules should block.
- Data leakage
- A model or application revealing data it should not, such as personal data, secrets, training data, or its system prompt.
- Multi-turn attack
- An attack spread across several conversation turns, where each step looks harmless but the sequence produces a failure.
- Tool misuse
- An agent being manipulated into calling a tool or API in a way its owner did not intend, such as moving data or taking an unauthorized action.
- MCP (Model Context Protocol)
- An open protocol that connects AI agents to tools and data sources. MCP servers are a common target in agent testing.
- OWASP Top 10 for LLM Applications
- A community list of the most important security risks for applications built on large language models, used to categorize findings.
- MITRE ATLAS
- A knowledge base of adversary tactics and techniques against AI systems, with technique IDs that red-team findings can reference.
- NIST AI RMF
- The US National Institute of Standards and Technology’s AI Risk Management Framework, often used to organize AI risk reporting.
- Guardrails (runtime protection)
- Controls that inspect and block unsafe inputs or outputs while an AI application runs in production. Some vendors turn red-team findings into guardrail rules.
- Automated red teaming
- Red teaming where software, often an attacker model or agent, generates and runs attacks against a target without a person writing each test.
- Retest
- A follow-up test that checks whether a reported finding was fixed.
- Regression testing
- Re-running known successful attacks after every model, prompt, or code change to confirm old issues have not come back.
- Rules of engagement
- The written agreement on what testers may attack, when, with which data, and what they must not do.
- SOC 2 Type II
- An independent auditor’s report on how a company’s security controls operated over a period.
- ISO/IEC 27001
- An international certification for an information security management system.