Red-teaming an AI system means deliberately trying to make it behave badly, before shipping, rather than discovering the failure modes from real users after launch. It's broader than the prompt injection testing covered elsewhere on this site — that's one specific attack category. Red-teaming also probes for the system producing harmful, biased, or simply embarrassing outputs under adversarial or unusual input, and for the system being manipulated into behaviour outside its intended scope through means that have nothing to do with hidden instructions in content.
The exercise works best with people who weren't involved in building the system, because the team that built it has an internalised sense of how it's meant to be used and unconsciously avoids the inputs that would break it. Someone approaching it fresh, actively trying to make it fail rather than trying to demonstrate that it works, finds problems the builders structurally cannot see in their own system.
Cover the categories systematically rather than testing randomly. Can the system be convinced to ignore its stated constraints through persistent or creatively framed requests? Does it produce meaningfully different quality or tone of response for different demographic framings of the same underlying question, revealing a bias that wasn't visible in aggregate testing? Can it be led into generating something the product should never produce — instructions for harm, confidently stated false information presented as fact — through a sufficiently indirect path?
Document what's found and, critically, what remains unfixed and why, rather than treating red-teaming as a one-time exercise before launch that's then forgotten. Models get updated, prompts get modified, new features get added that change the attack surface — a red-teaming exercise from launch tells you nothing about the system's current state eight months and eleven feature releases later. Building it into the release process for anything touching the AI system, the same way security testing is built into a deployment pipeline, is what keeps the exercise actually protective rather than a one-time compliance box ticked before launch.
