How to choose an AI red teaming vendor

01

Define the target and the risks

• What you are testing: a base model, a chat application, a RAG system, or an agent that calls tools and APIs
• Security risks: prompt injection, indirect prompt injection, jailbreaks, data leakage, system prompt extraction
• Agent risks: tool misuse, privilege escalation, goal manipulation, MCP server abuse
• Safety and policy risks: harmful content, off-policy answers, regulated advice
• Environment: staging or production, API or user interface, and which data the tester may touch

Write one sentence on what “done” looks like: which systems, which risk categories, and what evidence closes a finding.

02

Choose a buying model

• Managed engagement: human researchers run a scoped, time-boxed test and deliver a report. Fits pre-launch reviews and audits.
• Crowdsourced researchers: a vetted crowd tests your system, often with a platform for triage. Fits broad, creative coverage.
• Automated platform: the vendor’s attack agents run against your endpoint on demand or on a schedule. Fits continuous testing and CI/CD.
• Open-source tooling: your engineers run the tests. Lowest cost to start; your team owns scoping and triage.

Many teams combine one human-led engagement before launch with an automated platform for regression testing afterward.

03

Check coverage against frameworks

Ask which OWASP Top 10 for LLM Applications categories and which MITRE ATLAS techniques the vendor tests, and whether each finding is tagged with those IDs. Ask for the attack categories in writing, including multi-turn and multimodal attacks if those matter to you. Attack-library sizes are vendor figures; a list of what is tested against your system is more useful than a count.

04

Test agents and tool use, not only prompts

If your system calls tools, reads retrieved content, or connects to MCP servers, confirm the vendor tests those paths: indirect prompt injection through documents and web pages, tool misuse, privilege escalation across tools, and memory or state across sessions. Ask how the tester gets access to tools safely and whether they test through the real user interface or only the API.

05

Agree on deliverables

Ask for a sample report before you sign. Useful reports include an executive summary, severity-ranked findings, reproducible transcripts or attack traces, framework mapping, and a remediation roadmap. Agree on the format engineers need (for example tickets or raw data) and on who owns findings and test data after the engagement.

06

Plan retesting

Ask whether a retest after remediation is included, how long the retest window is, and whether successful attacks are kept as regression tests. For platforms, ask how tests run in CI/CD or on a schedule, and how results are compared across model or prompt changes. Some vendors include retesting only in higher tiers.

07

Vendor evidence, ownership, and commercials

Ask for current SOC 2 Type II or ISO/IEC 27001 reports and their scope, since a certification may cover a parent company or one product only. Check data handling for prompts, transcripts, and any production data the testers see. Several vendors in this market are now owned by larger security companies or have pending acquisitions, so ask about roadmap, contract entity, and continuity. Clarify the pricing unit (per engagement, per application, per seat, or per probe), retest pricing, and termination terms.