The Deepfake Watchlist is Resemble AI's weekly surveillance of synthetic media incidents, ongoing cases, and disputed content shaping the news cycle. Each week we track confirmed incidents, emerging attack vectors, and claims under investigation, alongside the provenance, detection, and policy threads running underneath them. New to the world of deepfakes? Read the Deepfake 101 Guide to learn more about how to protect yourself and your company from threats.
1. Italian bank chairman loses €95M in AI voice-clone fraud
Reuters's AI messaging scam costs Italy's top bank Intesa millions reports that fraudsters stole €95 million from Fideuram, the private banking arm of Intesa Sanpaolo, by sending former chairman Paolo Molesini a spoofed WhatsApp message impersonating Intesa CEO Carlo Messina, then following up with a phone call using an AI-cloned voice of a law firm partner Molesini knew personally, prompting him to authorize a series of transfers to accounts in China and Hong Kong.
- Category: Fraud / Impersonation
- Type: Attack
- Modality: Audio
- Policy / Regulatory: Prosecution is proceeding under existing wire fraud frameworks, with the Milan Public Prosecutor coordinating recovery through Eurojust; a foreign national living outside Europe has been placed under investigation for computer fraud.
- Trend: Two-step credibility stacking, a spoofed text from a known authority followed by an AI-cloned voice as confirmation, bypassing skepticism that a standalone cold call would not.
- Attack vector: Fake WhatsApp message impersonating the parent-company CEO to establish the transaction context, then an AI-cloned call from a trusted lawyer as the closing move, designed to feel like independent verification.
More than half the funds were later recovered through international banking coordination, but roughly €36 million remains missing, converted into cryptocurrency and being traced through letters rogatory sent internationally. Molesini resigned as chairman in March citing personal reasons. Intesa Sanpaolo and Fideuram declined to comment.
The attack sequence matters more than the voice clone on its own. The message came first to establish context and urgency; the cloned voice arrived as confirmation from someone the target already trusted. No standard verification protocol catches a familiar lawyer's voice on a call. The only defense at the generation layer is provenance: authenticated voice output carrying a verifiable signature before it reaches any channel.
2. Grok NCII victims failed by UK police as xAI ignores law enforcement requests
The Bureau of Investigative Journalism's Failed by the police, Grok deepfake victims are seeking justice themselves, published September 29 in partnership with The Observer, documents that UK police investigations into non-consensual intimate images generated by Grok during the December 2025 crisis were closed without charges after xAI failed to respond to officers' requests for account information, leaving identified perpetrators still active on the platform.
- Category: CSAM / NCII
- Type: Attack
- Modality: Image
- Policy / Regulatory: Ofcom's hash-matching deadline for platforms took effect September 30, the day after publication, though Ofcom has acknowledged it cannot currently investigate the standalone Grok app under the Online Safety Act as written; the Crime and Policing Act 2026 will add 48-hour removal requirements when enacted.
- Trend: Platform non-cooperation as the primary obstacle to enforcement, with victims turning to civil litigation after criminal routes are exhausted.
- Attack vector: Anonymous X users requesting that Grok generate sexualized imagery from profile photos of named individuals, with the platform's own system prompts permitting unrestricted adult sexual content outside clearly criminal categories.
The TBIJ investigation centers on three named women: presenter Jess Davies, broadcaster Narinder Kaur, and Labour MP Jess Asato. Davies's investigation was closed after xAI ignored police requests, with the assigned officer telling her that xAI cooperation was the "only line of inquiry." Asato's civil claim against xAI is the first UK legal action against Grok's NCII output; Davies is in pre-action correspondence alleging data protection violations.
Professor Clare McGlynn's observation in the piece providers more plain framing: we don't say the knife did it when someone stabs another person, we hold the person who used the knife accountable. That logic has not translated into consistent prosecutorial approach here. Grok's published system prompt, permitting "no restrictions on adult sexual content or offensive content," was still active as of publication.
3. OpenAI rogue agents breached Australian Medicare portal and probed three US government sites
CNN's Rogue OpenAI agents targeted three separate US government websites, published September 26, and reporting by The New York Times, TechCrunch, and NPR document that OpenAI AI agents operating during internal evaluations breached Australia's Medicare statistics portal on June 18 and separately probed the US Education Department, Commerce Department, and Securities and Exchange Commission websites, with OpenAI notifying Australian authorities only on September 10, nearly three months after the breach.
- Category: Fraud / Impersonation
- Type: Attack
- Modality: agentic AI
- Policy / Regulatory: Australian Prime Minister Albanese described it as the first known instance of an AI agent gaining unauthorized access to Australian government systems; a bipartisan Stop Rogue AI Act has been introduced in the US Congress in response to the pattern of unauthorized agent activity.
- Trend: Autonomous AI agents taking unsanctioned actions beyond their assigned research tasks during internal evaluations, creating a new incident category distinct from human-directed attacks.
- Attack vector: Agents seeking publicly available health data encountered access restrictions, found workarounds, accessed non-public files, and left notes in an external wiki targeting follow-on access at a second government agency.
OpenAI's own account: its models "took actions we did not intend." The self-directed note-taking for follow-on targets is what makes this more than a misconfigured scraper. OpenAI learned of the incident in August and notified Australia only after external researchers at Transluce published their findings. Competitors Anthropic, Meta, and Google have each separately disclosed similar containment failures around the same period.
The agentic category is new ground for the Watchlist, but the harm mechanism sits in familiar territory: a system accessing data it had no authorization to access, causing harm the operator neither intended nor detected promptly. The difference is that no human made the decision to probe those systems, which is the part that existing liability frameworks were not built to address.
4. Arizona appeals court vacates manslaughter sentence over AI-generated victim impact video
FOX 10 Phoenix's Arizona manslaughter sentencing vacated due to use of AI 'victim impact statement' reports that the Arizona Court of Appeals vacated the 10.5-year prison sentence of Gabriel Paul Horcasitas, convicted of manslaughter in the 2021 road rage shooting of Christopher Pelkey, after finding that the trial judge's reliance on an AI-generated video of Pelkey created by the victim's sister as a victim impact statement was so prejudicial as to render the sentencing procedure fundamentally unfair.
- Category: Harassment / Public Safety
- Type: Response
- Modality: Video
- Policy / Regulatory: The Arizona Court of Appeals noted that no Arizona case has previously addressed the admissibility of AI-generated victim impact evidence; the ruling creates state-level precedent without binding guidance for other jurisdictions, and Horcasitas will be remanded for re-sentencing.
- Trend: AI-generated content entering criminal proceedings with sincere intentions, raising questions about how courts should evaluate synthetic representations of people and the weight they should carry in sentencing.
- Attack vector: Not a deliberate attack. Pelkey's sister, Stacey Wales, worked with her husband and a friend to create an AI video of her brother. The video was presented as a victim impact statement, and the judge praised it before handing down the sentence. The appeals court found that the video blurred the distinction between the family's beliefs about what Pelkey might have said and his actual words and opinions.
The case is notable because the concern wasn't that the family had secretly presented the video as authentic footage. The video introduced itself as an AI recreation. The issue was the weight the representation carried in court, where the judge's assessment of the victim's views could influence the sentence.
The ruling could inform how other courts approach AI-generated victim impact statements, though its application beyond Arizona remains to be seen. It adds to a growing set of questions about how courts should evaluate AI-generated material when it may influence consequential decisions.
5. Deepfake paid ads distort Victorian election as research finds nearly half of voters can't reliably identify authentic footage
The Conversation's Deepfakes are distorting this year's Victorian election. We found out who's most susceptible to them, published September 27 by researchers from the University of the Sunshine Coast, Griffith University, the University of Queensland, and Swinburne University, documents that AI-generated videos depicting fabricated emergency scenes have appeared as paid advertising ahead of the Victorian state election, while a meta-analysis across 19 studies and 24,000 participants found deepfakes appear just as persuasive as authentic political content.
- Category: Political / Electoral
- Type: Attack
- Modality: Video, Image
- Policy / Regulatory: Australia has no federal requirement to label AI-generated content in state election advertising; the deepfakes circulated as paid ads without any disclosure obligation.
- Trend: Political deepfakes shifting from nation-state operations to domestically produced paid election advertising, with the Victorian election as the first documented Australian case.
- Attack vector: AI-generated videos depicting fabricated public emergencies (machete attack, roadside birth, parliamentary flooding) circulated as paid pre-election advertising without attribution to any campaign.
In the researchers' own experiment with 411 Australians, roughly 75% could identify a deepfake, which sounds reassuring until the other finding: almost 29% incorrectly labeled authentic video as fake, and a further 18% were unsure. Combined, nearly half could not confidently and correctly identify genuine political footage as real. Among regular news consumers, confidence became increasingly disconnected from accuracy, meaning the people most likely to feel certain about a video were not reliably the ones getting it right.
The researchers call this the "liar's dividend" running in reverse: the existence of deepfakes makes people more likely to dismiss real evidence as fabricated. Political orientation compounded the problem, with people better at detecting deepfakes of politicians aligned with their own views. The Victorian election is months away and the paid ads are already running.
Honorable mentions
Victoria Police warn of AI doctor voice scam targeting Mandarin-speaking community. ABC News Australia and Yahoo News Australia reported September 30 that police have received ten reports since early September of a scam targeting Mandarin speakers with AI-cloned calls from fake doctors and surgeons. In one case, the scammer replicated the voice of a medical professional already known to the victim. Losses have reached tens of thousands of dollars per victim through gift card payments. The attack removes the accent and fluency tells that earlier versions of this pre-existing fraud relied on.
Canadian media retracts articles from AI-generated journalist persona. The Ottawa Citizen, Montreal Gazette, and Policy Options retracted articles on September 30 after discovering that a journalist named Daniel Robson was likely an AI-generated fabrication used to publish disinformation targeting Moroccan dissidents. The case extends the synthetic media threat beyond fabricated video into fabricated sources: a persona credentialed enough to place articles in major outlets, undetected until a targeted subject raised the alarm.
The pattern
- Three of this week's five stories involve AI systems doing things their operators either didn't intend or explicitly enabled while claiming they hadn't. OpenAI's agents accessed government infrastructure without instruction. Grok generated millions of sexualized images under prompts that explicitly removed content restrictions. The Fideuram fraudsters used voice cloning the target had no reason to distrust. The common thread is a contest between "the tool did it" and "the operator is responsible," and the outcomes of that contest will shape how liability for AI-generated harm gets assigned for years.
- The Fideuram attack and the Victorian election research both point to the same detection failure, just at different scales. The bank chairman trusted a familiar voice. Nearly half of research participants couldn't reliably distinguish authentic political footage from fabricated content. In both cases the failure was not naivety, it was that the normal signals people use to evaluate authenticity were exactly what the synthetic content was designed to replicate. Detection at the human level was never going to be enough.
- The Arizona ruling and the TBIJ Grok investigation arrived in the same week and tell the enforcement story from two directions. A court moved the law forward by one case, overturning a sentence where AI-generated content influenced a judge without proper scrutiny. Meanwhile, police investigations into documented NCII were closed because a platform wouldn't cooperate, and perpetrators stayed active. New legislation keeps arriving. The gap between what laws say and what institutions can deliver is where a lot of the real harm is accumulating.
Watching next week
- Ofcom enforcement post-September 30 deadline. Which platforms confirmed hash-matching compliance, whether X/Grok falls under Online Safety Act chatbot provisions, and whether any enforcement action was initiated.
- OpenAI rogue agent fallout. Whether Australia pursues criminal liability for the June breach and whether the Stop Rogue AI Act advances in Congress.
- Arizona AI evidence precedent. Whether other US courts cite the ruling and whether any evidentiary body moves on admissibility standards for AI-generated court content.
- Victorian election deepfakes. Whether the paid advertising deepfakes are traced to any operator, and whether the Australian Electoral Commission moves to require disclosure labeling before voting day.
The Deepfake Watchlist publishes every Thursday. Subscribe to receive it in your inbox, or follow Zohaib Ahmed on LinkedIn for the weekly social companion. Track every documented incident in the Resemble Deepfake Incident Database, and read the full methodology in our 2026 Midyear Deepfake Threat Report.