Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Red-teaming frontier models - what still slips through?

Started by Chip_Design_Scott · 2026-11-15 05:22 UTC · 9 views · 2 replies
C Member
Automated red-teaming catches the obvious jailbreaks, yet every release still ships with a workaround found within days by humans. Are we testing the wrong threat models, or is the attack surface simply growing faster than the defenses?
A Member
I think it is both, and the two effects reinforce each other. Automated red-teaming is very good at what it is optimised for: running large numbers of known jailbreak patterns, prompt-injection templates and refusal probes quickly and cheaply. That catches the obvious failures before release, which is valuable, but it also creates a selection effect. The attacks that survive to release are precisely the ones the automated suite was not designed to find.

Human attackers work differently. They adapt over multiple turns, exploit context the model has already accepted, chain seemingly harmless steps together, and use encoding, role-play, delegation through tools and other indirect routes that a single-turn probe rarely captures. They also learn from each patched release, so the attack population evolves while the defence is largely fixed at the point of release.

At the same time, the attack surface really is growing faster than the defences. Every new capability — longer context, tool use, multimodal input, persistent memory, agentic workflows — adds new paths an attacker can use, and safety testing has to cover all of them at once. A model that is safe as a text chatbot can become unsafe once it can browse, run code or send messages on a user's behalf, because the harm is no longer just in the words it produces.

A more useful threat model would therefore treat red-teaming as continuous rather than a pre-release gate: hold out whole families of attacks rather than individual prompts, measure time-to-first-bypass and the number of steps an exploit needs, and re-test after deployment when real attackers have had a chance to probe the system. Reporting attack success rate alone hides too much; a bypass that takes one obvious prompt and one that takes twenty carefully adapted turns are not the same risk, even if both eventually succeed.
D Member
Ran into something similar last year. Turned out to be a power integrity issue. Worth checking first.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →