AI Watermarking Weakens Safety Guardrails in Language Models
Researchers found that SynthID-Text watermarking, deployed by major AI providers to comply with EU AI Act requirements, unintentionally weakens language model safety guardrails. The watermarking alters how models select tools and respond to harmful prompts, with the vulnerability growing sharper under prompt injection attacks.
What happened
Lasso Security researchers tested Google DeepMind's SynthID-Text watermarking scheme, which is being deployed in models from Anthropic and OpenAI. The watermarking system, designed to mark AI-generated content for traceability under EU AI Act requirements, alters token probability distributions during generation.
Testing on seven language models using the BFCL v4 tool-calling benchmark found that watermarking reduced tool selection accuracy in six of seven cases, with four showing significant drops. The effect was amplified under prompt injection attacks, where weakened safety refusals became more vulnerable to adversarial inputs.
Reported impact
Watermarking introduces sampling drift that changes model outputs and agent behavior. At the model level, safety refusals become unpredictable; at the agent level, watermarking influences which tools are called and with what parameters.
Individual prompts produced different tool choices or argument values between watermarked and plain runs, even when overall accuracy scores appeared similar. Developers of AI agents built on these models must retest safety and accuracy after watermarking deployment.
Index analysis
The research reveals an unintended trade-off in regulatory compliance. Watermarking mechanisms designed to satisfy transparency requirements inadvertently weaken the technical controls intended to prevent harmful outputs.
The vulnerability grows more severe when adversaries combine watermarking effects with prompt injection techniques, as the reduced refusal rates become exploitable through API access and external function calls in autonomous agent deployments.
Governance controls
EU AI Act requirements drove widespread adoption of statistical watermarking across major model providers. However, the testing reveals that compliance mechanisms introduce safety degradation that regulators and developers did not anticipate.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.