The Deepfake Watchlist is Resemble AI's weekly surveillance of synthetic media incidents, ongoing cases, and disputed content shaping the news cycle. Each week we track confirmed incidents, emerging attack vectors, and claims under investigation, alongside the provenance, detection, and policy threads running underneath them. New to the world of deepfakes? Read the Deepfake 101 Guide to learn more about how to protect yourself and your company from threats.
1. A fake Mitch McConnell photo got caught, then a real one got wrongly accused
Snopes's Fake Mitch McConnell hospital image spreads as questions surround his health confirmed that a viral image showing Senator Mitch McConnell hooked to tubes in a hospital bed was AI-generated, detecting Google's SynthID watermark in two versions of the photo circulating on Reddit and X.
- Category: Political / Electoral
- Type: Attack
- Modality: Image
- Policy / Regulatory: No congressional or platform standard currently requires health-related images of sitting officials to carry verifiable provenance markers before they circulate.
- Trend: A sitting official's unexplained hospitalization is becoming reliable bait for fabricated content, since public uncertainty creates exactly the information vacuum synthetic images are built to fill.
- Attack vector: A single fabricated image exploiting weeks of genuine, unexplained absence from public view to make speculation feel confirmed.
McConnell's office had stayed quiet about the specifics of his hospitalization for weeks, and that silence is what gave the fake image room to spread. Reddit and X users had no real information to check it against, just a senator who had genuinely disappeared from public view since mid-June, and SynthID confirmed the fake once Snopes went looking.
The story should have ended there, but days later McConnell's office released an actual photo to address the speculation, and it was immediately met with accusations that it, too, was AI-generated, amplified by Grok, which hallucinated a fact-check source that never existed. PolitiFact and Snopes had to debunk the debunking. Watermarking correctly cleared the real photo and correctly caught the fake one, but by that point the public argument wasn't really running on evidence anymore.
2. xAI sues Grok user over CSAM deepfakes
CNN's Elon Musk's xAI sues user over allegedly creating child sexual abuse materials with Grok reports that xAI filed a federal lawsuit against South Carolina resident Terry Harwood, alleging he used deceptive, repeatedly rewritten prompts to bypass Grok's safeguards and generate sexually explicit deepfakes of minors from uploaded photographs.
- Category: CSAM / NCII
- Type: Response
- Modality: Image
- Policy / Regulatory: This is among the first times an AI company has sued its own user civilly over CSAM generated on its platform, rather than relying solely on criminal referral.
- Trend: AI companies are increasingly using civil suits against individual users to draw a line between platform misuse and product liability.
- Attack vector: Iteratively rewritten prompts designed to work around content-moderation refusals until the system produced explicit material.
What makes this suit notable is who is doing the suing, and what it leaves unanswered. xAI is going after its own user in civil court instead of leaving the matter to prosecutors, a posture critics say shifts legal responsibility from the AI system onto the person who defeated its safeguards, rather than the company answering for why those safeguards could be defeated in the first place. The complaint states the company suspended more than 52,000 accounts and made over 73,000 reports to the National Center for Missing & Exploited Children in 2026 alone, a scale that says more about the size of this problem than any single lawsuit can.
The Harwood case doesn't sit in isolation either. xAI is separately facing a lawsuit from the city of Baltimore over Grok's "Spicy Mode" generating millions of sexual images, and a class action from women whose childhood photos were scraped and turned into explicit content by the same product. Suing individual users is one way to respond to a pattern like that. It isn't obviously the most important one.
3. Meta pulls Muse Image, then its own detector fails a basic test
Bangkok Post's Meta halts Instagram AI image feature after fierce backlash published that Meta discontinued its Muse Image tool on Instagram on July 10, three days after launch, following backlash over a default opt-in setting that let anyone generate images using public accounts as a reference.
- Category: Brand / Likeness
- Type: Response
- Modality: Image
- Policy / Regulatory: Meta's own Oversight Board called on the company in March to invest in stronger detection tools for AI-generated content.
- Trend: Detection tools built around invisible watermarking keep proving fragile against ordinary edits, cropping, screenshotting, and compression.
- Attack vector: An opt-in-by-default feature that let any user generate synthetic images referencing another person's public profile without consent.
The consent failure came first, and it's what actually caused the backlash. Meta shipped a tool that treated every public Instagram account as fair game for AI reference images unless the account holder found the toggle and turned it off.
Meta had previewed a detection tool built around an invisible watermark called Content Seal, marketed as durable through cropping and compression. Reuters tested that claim, and the tool missed more than half of its own images once resized down, a reminder that provenance has to survive the ordinary edits people make before sharing.
4. AI voice clones drain billions from Medicare
The New York Post's Criminals scamming billions from Medicare using fake AI voices, hacking data, all without setting foot in the USA found that international criminal networks are using AI voice cloning and hacked patient data to defraud Medicare and Medicaid of billions of dollars, without any of the people running the schemes ever setting foot in the United States.
- Category: Fraud / Impersonation
- Type: Attack
- Modality: Audio
- Policy / Regulatory: CMS enforcement has leaned heavily on data analytics to flag suspicious billing, but voice-based social engineering against IVR systems remains a gap most healthcare fraud enforcement wasn't built to cover.
- Trend: Fraud rings are running these schemes at industrial, automated scale rather than one call at a time.
- Attack vector: Cloned voices used to authorize transactions and manipulate human agents through high-volume, automated calling campaigns.
This is worrisome for anyone who has ever trusted a phone call from someone claiming to represent a program they rely on. The scale here is an automated pipeline built to probe thousands of accounts, bringing in a human scammer only once the system has done the initial work.
The mechanics matter as much as the dollar figure. Fraudsters run voice bots against IVR systems first, the same automated phone menus everyone navigates daily, before handing off to a human operator, closer to how credential-stuffing attacks work against a website than to a single scammer working the phones. None of this requires deceiving a person into believing they're talking to someone they know, which is what makes it distinct from the grandparent scams that get most public attention.
5. AI crime-alert apps are generating their own false alarms
KXXV's Waco police warn residents about AI-generated crime alerts that may spread false information reports that Waco police are urging residents to verify AI-generated crime alerts after an app that processes police scanner audio merged two separate channels and falsely reported that an officer had been shot, when the audio it misread actually came from a training exercise.
- Category: Harassment / Public Safety
- Type: Attack
- Modality: Audio
- Policy / Regulatory: No AI-generated crime or travel alert app currently carries a verification standard close to what applies to licensed emergency broadcast systems.
- Trend: This wave of public-safety confusion doesn't need a deliberate deepfake at all, an AI system misreading raw audio or fabricating an article is enough on its own to trigger a real-world response.
- Attack vector: Automated systems processing raw scanner audio or generating unverified content and pushing it to the public without a person checking it first.
Waco PD's explanation was refreshingly plain, an app took back-channel and main-channel scanner traffic, merged them, and produced a report that an officer had been shot when nothing of the kind had happened. Nobody built this to deceive anyone, which is what makes it a different problem than the deepfakes elsewhere in this issue.
The same week, Toronto Pearson was fighting a version of the same problem from the other direction, sites publishing dozens of AI-written articles a day inventing flight disruptions with no attempt at verification. Akron's own false active-shooter report days later suggests this is a category of failure showing up in multiple cities the same week, one where the accuracy of the underlying AI is a public-safety question in its own right, not just a content-moderation one.
Honorable mentions
A few other stories from the week worth a shorter note.
FPL warns of AI-generated scam calls targeting utility customers, NBC Miami. Florida Power & Light is warning customers about AI voice calls promising up to $4,000 in fake energy rebates, using urgency language like "last notice" to push people toward a scam number.
The pattern
.jpg)
- Detection did its job this week and still couldn't fully settle the argument. McConnell's fake hospital photo was caught by watermarking, and days later a real photo of him was wrongly accused of being fake too, the tools got both calls right, but by then the public conversation wasn't running on evidence anymore. Meta's cropping test showed the same gap from another angle, provenance holds up in a clean test and frays once content moves through the ordinary edits people make before sharing.
- Not every incident this week required a bad actor. Waco, Toronto Pearson, and Akron all produced false public-safety information from AI systems that weren't trying to deceive anyone, they just got the underlying audio or content wrong and nobody caught it before it went out. That's a different risk category than a deliberate deepfake, and it deserves its own scrutiny.
- Platforms are starting to hold individual users legally accountable for weaponizing generative tools, as xAI's suit against Terry Harwood shows, while continuing to build products, like Muse Image, whose defaults created exactly the kind of harm those legal actions are meant to punish. August's EU AI Act Article 50 enforcement deadline could start forcing platforms to answer for that inconsistency.
Watching next week
- EU AI Act Article 50 enforcement. Transparency obligations for AI-generated content land in August, and this week's Meta and xAI stories are a preview of the accountability questions regulators will be asking.
- Sports deepfakes through the rest of the World Cup. If this pattern held for one week, it's worth watching whether it escalates as the tournament reaches its final rounds.
- Whether Meta reintroduces Muse Image with an opt-in default. The tool is paused, not killed, and how Meta redesigns consent will say a lot about whether this was a lesson learned or a pause for the news cycle to pass.
- Follow-up on the xAI-Harwood suit. It's an early test of whether civil litigation can meaningfully deter users who weaponize generative tools.
The Deepfake Watchlist publishes every Friday. Subscribe to receive it in your inbox, or follow Zohaib Ahmed on LinkedIn for the weekly social companion. Track every documented incident in the Resemble Deepfake Incident Database, and read the full methodology in our 2025 Deepfake Threat Report.
