The Deepfake Watchlist is Resemble AI's weekly surveillance of synthetic media incidents, ongoing cases, and disputed content shaping the news cycle. Each week we track confirmed incidents, emerging attack vectors, and claims under investigation, alongside the provenance, detection, and policy threads running underneath them. New to the world of deepfakes? Read the Deepfake 101 Guide to learn more about how to protect yourself and your company from threats.
1. AI chatbot's false nuclear cargo report nearly triggers US boarding of a Chinese ship
Gulf News's AI-generated report nearly triggered US operation on Chinese ship reports, citing CNN, that a US Special Operations Command Pacific analyst asked a chatbot to fuse open-source information with classified signals intelligence on a Chinese vessel's manifest this spring, and the system hallucinated a nuclear weapons component classification that circulated through military channels until officials caught the error just before armed troops boarded the ship.
- Category: Harassment / Public Safety
- Type: Attack
- Modality: Text (AI-generated intelligence output, no video, image, or audio synthetic media involved)
- Policy / Regulatory: No established framework governs independent verification of AI-assisted intelligence products before operational action, leaving human sign-off practices to individual command discretion.
- Trend: Fabricated but authoritative-sounding AI output entering high-stakes decision chains, extending deepfake-style harm beyond video and audio into text-based intelligence products.
- Attack vector: An analyst tasked a chatbot with reading a ship's cargo manifest, the chatbot hallucinated a nuclear-cargo classification, and the output was formatted into a standard intelligence report and passed along without independent verification while military planes were already in the air.
There's no cloned voice or fabricated video here, but the mechanism is the one we track every week: an AI output polished enough that nobody reading it stopped to question it. A military source told CNN that AI in targeting is ramping up with no real guidance for keeping a human meaningfully in the loop, and put the deeper risk bluntly: AI lets you get to a bad idea faster.
2. Federal judge blocks Montana's AI deepfake political ad law over free speech concerns
States Newsroom's Federal judge says AI deepfake election law violates First Amendment reports that US District Judge Susan Watters issued a preliminary injunction on September 18 protecting a political action committee that used AI-altered candidate images in campaign mailers, finding Montana's deepfake disclosure statute likely violates the First Amendment's protection of political speech.
- Category: Political / Electoral
- Type: Response
- Modality: Image
- Policy / Regulatory: Montana's Senate Bill 25 criminalizes AI-generated deepfakes in campaign material with civil fines and up to two years in prison; the injunction blocks enforcement against the named plaintiffs only and does not repeal the statute.
- Trend: Courts treating AI disclosure mandates for political speech as content-based regulation subject to strict scrutiny, continuing a pattern of First Amendment skepticism toward state-level deepfake election laws.
- Attack vector: Not applicable, this is a judicial response to a free speech challenge rather than an attack incident.
Watters wrote that the court lacked authority to issue a broader injunction covering every speaker, so the ruling protects this one PAC while the underlying law stays on the books for now. What's actually being tested here isn't whether the mailers misled anyone, but whether a state can compel disclosure of AI involvement in political content without bumping into the same protections that cover satire and caricature.
It sits next to the 7th Circuit's CSAM ruling from a few weeks back as another data point in a pattern that keeps showing up this year: courts finding that existing constitutional frameworks weren't built with AI generation in mind, and legislatures reaching for fixes that run into walls they didn't see coming.
3. Commonwealth Bank deploys AI agents to counter deepfake investment scams
Mortgage Professional Australia's CBA deploys AI agents to fight back against deepfake scams reports that new Commonwealth Bank research finds 89% of Australians believe they could identify an AI-generated deepfake scam, but only 42% actually did when tested, while the bank's "Pollen Team" of AI agents has engaged scammers in more than 350,000 automated conversations since launching last year.
- Category: Fraud / Impersonation
- Type: Response
- Modality: Video, Audio
- Policy / Regulatory: No new statute here; CBA points to industry coordination through the Scam-Safe Accord and the Fraud Reporting Exchange as the collaborative backbone behind faster cross-institution intervention.
- Trend: Banks turning AI defensively against AI-enabled scams, and pushing prevention efforts upstream to the credibility-building stage rather than the moment money actually moves.
- Attack vector: Criminals use deepfake video and cloned voice to impersonate trusted brands and public figures, building credibility on social platforms and messaging apps before moving victims toward a payment.
James Roberts, CBA's executive general manager for fraud and scams, put the timing problem plainly: by the time someone is making the payment, the deception has already succeeded. A warning at the transaction screen arrives far too late for a victim who has spent weeks building trust with a synthetic persona, so the bank is pushing intervention back to that earlier stage instead. Its Fraud Detection Agent reviews millions of transactions daily and has helped update roughly 3,000 fraud rules, drawing on intelligence the Pollen Team's conversations with scammers generate.
Nine in ten Australians in CBA's survey were sure they'd catch a deepfake scam, but only four in ten actually did when tested. Criminals are counting on exactly that gap between confidence and actual detection ability.
4. Meta ordered to remove deepfakes targeting women politicians as Oversight Board calls safeguards "fundamentally inadequate"
PetaPixel's Meta's Oversight Board Orders the Company to Remove Deepfake Videos From Facebook reports that Meta's Oversight Board overturned the company's original decisions and ordered removal of two AI-generated videos, one depicting a Scottish Labour councillor making inflammatory anti-refugee comments she never made, and one falsely showing a Muslim health campaigner giving absurd advice while eating junk food.
- Category: Harassment / Public Safety
- Type: Response
- Modality: Video
- Policy / Regulatory: The board's case decisions are binding on Meta, and it issued nine broader policy recommendations that are not binding, with Meta required to respond within 60 days.
- Trend: Independent oversight bodies are formally documenting that platform deepfake safeguards, tuned mainly for electoral disinformation, fail when the target is a private or non-electoral public figure being harassed rather than a candidate near a vote.
- Attack vector: An AI-generated video with audio not synchronized to facial movements was posted to Facebook, and Meta's automated systems did not escalate it for human review even after two reports, including one from the councillor herself.
Meta's own reviewers had cleared the video because she is an adult public figure and the claim was framed as an assertion about some refugees rather than all of them. That reasoning held up fine on a policy checklist, and it did nothing about the actual harm of a fabricated hate-speech clip attributed to a real woman. The board called the underlying rules "fundamentally inadequate" for how deepfakes are actually being used.
Oversight Board co-chair Pamela San Martin tied both cases to a wider pattern, in which women who speak publicly are disproportionately targeted with fabricated content built to discredit or silence them. Read together, the two cases look less like isolated incidents and more like a harassment tactic with its own infrastructure behind it.
5. DHS expands identity verification testing to target deepfakes and AI-generated IDs
Biometric Update's DHS expands RIVR testing to deepfakes, AI-generated IDs reports that the Department of Homeland Security's Science and Technology Directorate is adding a dedicated deepfake detection challenge to its 2026 Remote Identity Validation Rally, broadening document fraud testing after a 2025 evaluation found some commercial identity verification systems accepted more than three-quarters of fraudulent documents under worst-case conditions.
- Category: Fraud / Impersonation
- Type: Response
- Modality: Video, Image
- Policy / Regulatory: The expansion aligns with NIST's SP 800-63A-4 identity-proofing guidelines, finalized last year, which already require credential service providers to test against digital injection attacks and forged media; document validation applications open in October and the new deepfake category opens in December.
- Trend: Government-run adversarial testing programs are formalizing deepfake detection benchmarks after earlier rounds exposed dramatic performance gaps across commercial vendors.
- Attack vector: Not applicable, this is a testing and evaluation program rather than an attack incident.
One system accepted 76.68% of fraudulent documents under its worst tested conditions, and several passive liveness detection systems, the ones meant to catch a spoofed face without asking the person to do anything, misclassified 80% or more of attacks as legitimate. Only one document system and one liveness system stayed within DHS's own thresholds across every measure. The new deepfake category goes after exactly the gap those numbers exposed: synthetic impersonation in live video chats, not just static document forgery.
DHS will also start soliciting deepfake attack examples directly from organizations that hold threat data, beginning in November. It's an unusual step for a government testing program, and it says something about how far the attack landscape has outpaced what the agency's own labs can simulate on their own.
Honorable mention
It involves no synthetic media in the traditional sense, but it's worth flagging alongside this week's stories: Google says its Gemini model autonomously hacked three real companies during a cybersecurity test. The National's Google says Gemini model hacked three companies during test reports that during a May evaluation, Gemini found public information online and guessed credentials to access three real companies it mistook for in-scope test targets, stopping on its own each time. Google notified the affected organizations, and OpenAI, Anthropic, and Meta have each separately disclosed similar containment failures around the same period. Different mechanism, same underlying pattern: an AI system doing something its operators didn't catch until after the fact.
The pattern
- Two responses to AI-generated harm landed in the same week. A federal judge protected a political committee's right to distribute AI-altered campaign images, while an independent oversight body found a platform's deepfake safeguards fundamentally inadequate for protecting the people those images and videos target. Both are reaching for the same line, between expression and harm, and finding it in different places depending on whether the deepfake in question is aimed at a candidate or at a private citizen being harassed.
- Neither the SOCOM hallucination nor Gemini's unsupervised system access is a classic deepfake, but both trace back to the same root cause: verification lagging behind capability, so a plausible, well-formatted output gets more trust than it's earned. It's the same dynamic that makes a cloned voice convincing on a phone call, just showing up this time in an intelligence report and a sandboxed test environment.
- The defensive infrastructure built this week, CBA's scammer-facing AI agents and DHS's expanded deepfake testing program, is genuine progress, but the 2025 RIVR results are a reminder that testing has consistently lagged what generation tools can already do. DHS soliciting live attack data from outside organizations starting in November is essentially an admission that government labs can't keep pace alone, and other testing programs may end up following that same lead.
Watching next week
- Meta's 60-day clock. Whether Meta commits to concrete policy changes or a narrower fix by mid-November will say a lot about how seriously the "fundamentally inadequate" finding lands internally.
- Montana's underlying case. The injunction only covers the named plaintiffs, so watch whether the attorney general pursues enforcement elsewhere, and whether the constitutional question reaches a merits ruling.
- DHS's September 30 webinar. The informational session for prospective RIVR participants is the first concrete marker of how the expanded deepfake challenge will be structured.
- Further agentic AI disclosures. With Google, OpenAI, Anthropic, and Meta all now having disclosed AI systems escaping containment or accessing real systems, watch whether more labs come forward.
The Deepfake Watchlist publishes every Thursday. Subscribe to receive it in your inbox, or follow Zohaib Ahmed on LinkedIn for the weekly social companion. Track every documented incident in the Resemble Deepfake Incident Database, and read the full methodology in our 2026 Midyear Deepfake Threat Report.