Good intentions, bad consequences

Across the globe, governmental demand for a way to detect AI is greater than ever. Addressing world leaders at the UN General Assembly, last month Prime Minister Andy Burnham announced a new UK National Centre of Information Defence which will combat AI-amplified disinformation.

An active issue

A week after Burnham’s speech in New York, the US state of California legislated that being able to know where information comes from, including whether it’s AI, “is paramount to combating misinformation and disinformation as it relates to public health, election integrity, preventing fraud, extortion, or using [AI] to create content for other malicious purposes.” This Tuesday in his annual threat update, MI5 chief Sir Ken McCallum raised numerous risks posed by AI, including that “[it] can produce propaganda and radicalising material tailored to the individual.”

Marking and detecting AI, commonly referred to as watermarking, is not a new topic. The European Union’s Artificial Intelligence Act has given firms deadlines to change their world-wide offerings so AI content they produce can be detected. It covers text, images, audio, and video. California, the home of Silicon Valley, has passed similar requirements. A Code of Practice for doing this has been signed by most of the major AI companies including Anthropic, Google, Meta, and OpenAI. An approaching deadline is less than seven weeks away, at which point global industry standards will be cemented.

Concerningly, the detection method approved by policymakers is inherently flawed, yet due to a severe lack of public awareness, industry is on track to adopt it.

The risks

Unlike banknotes, whose watermarks we can see by holding them up to light, we won’t be able to see AI watermarks ourselves. Digital AI watermarks must be invisible to the public to prevent bad actors removing them. As a result, distinguishing between real and AI content will require exposing vast quantities of our authentic, non-AI content to AI companies and mass scanning servers, as only they can see if their watermarks are present. Scanned content must be handed over, fully accessible, as is explicitly permitted by the widely-signed Code of Practice. Nothing prevents this happening to our private communications.

A technique encouraged by the Code of Practice called signposting means it will be trivial for AI companies to filter the overwhelming majority of AI content from the real, human content they are sent – the exact valuable training data they need. Laws prevent our data being misused to benefit AI companies but aren’t realistically enforceable unless governments can check what they are doing with our data. Alternatively, AI firms could delegate this responsibility by collectively establishing a platform that performs these mass quantities of scanning, facilitated by “interoperability requirements” given to AI companies who have signed the Code of Practice.

Furthermore, AI watermarking policy will fail to detect a plethora of publicly available non-commercial AI models out in the open. It will also fail to detect AI models from foreign nations who do not trust us not to abuse access to their watermark detection tools, and who our governments do not trust to receive masses of our citizens’ data.

Why would our private content be affected?

If the platforms and devices we use choose or are required to label AI text, images, audio, and video, it means copies of things we type, photograph, and record will be sent away for scanning if they are deemed somewhere AI misinformation poses a risk. It doesn’t matter if our content isn’t AI – that can only be known once it has been sent away and scanned by every single AI company’s watermark detector.

Watermark detection by “hosting services”, “online platforms”, and by virtue of their similar nature, communication apps, is “strongly encouraged” by the EU, forming part of the AI content “value chain”. The factors in the law’s official motivation, “new risks of misinformation and manipulation at scale, fraud, impersonation and consumer deception,” apply both to things we interact with publicly and privately, including our secure communications. Remarks by MI5 chief Sir Ken bring light to the scope of these risks – “tailored to the individual” – further raising concerns about the scope of mitigations that may come. Motivating factors behind the world-first law were echoed by Burnham’s UN address with a focus on hostile foreign actors and mirrored by justifications in Californian legislation that include threats from other types of bad actors who may seek to deceive us.

A report published by the EU explains how the then-proposed law could mean social media and messaging apps perform detection, requiring content to be exposed, yet no laws were passed to make this impossible. In 2022, among others, Meta (the parent company of Facebook, Instagram, Messenger, WhatsApp), Google and TikTok signed an updated EU disinformation code in which they committed to proactively detecting AI content on their platforms. These commitments will soon be actionable. Returning to the UK, a July 2025 cross-party House of Commons committee report recommended that “All AI-generated content should be visibly labelled.”

Blank promises

Precedent-setting EU guidelines call for detection to be “secure, privacy-preserving and locally executable” whenever possible. This is fundamentally incompatible with AI watermarking. Watermarks must be invisible so they cannot be detected by ordinary people, otherwise bad actors would be able to evade them. This kills any possibility of local detection. A promise of privacy that relies on technology that does not exist is not privacy.

A glaring blind spot

News reports and AI company press releases have extensively covered how watermarks are added. However, rendered inaccessible by three-letter acronyms, hundreds of pages of legislation and dozens of sub-documents filled with tech jargon, the way watermarks are set to be used and detected has been starved the media attention it deserves. Deriving the implications requires technical knowledge about these systems, but feeling their effects will not.

In the absence of awareness of these issues, governments of countries such as the UK have every reason to adopt and follow flawed global and industry frameworks. These frameworks form international precedent, are in scope to cover mainstream AI providers’ worldwide offerings and come at no financial cost to governments. Passive approval of current policies will lead to their adoption and will expose us to the three core risks mentioned.

We must reject flawed AI detection methods that can only be used by sending troves of our real, human content to AI companies and mass scanning servers. We must recognise AI watermarking will fail to detect the AI models most likely to be used by bad actors. And most importantly, we must explicitly exclude non-public content from the scope of any future optional or mandatory AI labelling in order to maintain the integrity of our personal and private communications – a fundamental pillar of free speech and our democracy.