Trust + safety teams

Flag AI generated media before it reaches content moderation

Your existing classifiers identify content that violates platform policies. Resemble detects whether audio, video and images are AI-generated or manipulated, giving trust and safety teams another way to identify synthetic media.

Contact us
CASE STUDY

A major image licensing platform runs around two million images a month through verification before anything is published, and needs a verdict back inside a minute so editorial is not held up.

2M
images a month through one verification
queue

Where sythentic media gets on your platform

User generated uploads

Traditional user-generated content moderation was built for human users, not AI models that can churn out convincing fake content at scale. Creating fakes is cheaper than ever, and the volume keeps growing.

Impersonation and likeness abuse

AI-generated audio and video can make it look like real people, including public figures and the recently deceased, are saying or doing things they never did. The content may violate existing policies, but proving it’s synthetic can make enforcement difficult.

Coordinated campaigns

Fabricated profile photos, invented bylines and synthetic avatars used to manufacture accounts at scale. Social media content moderation catches individual posts while the pattern across accounts goes unnoticed.

Licensing and editorial verification

Marketplaces licensing content and newsrooms publishing it need to know the media is authentic. A single synthetic image, video or audio clip slipping through can undermine trust and credibility.

What your moderation stack catches, and what it misses

Your existing moderation tools are good at identifying content that violates platform policies. But policy violations and synthetic media are two different questions. Knowing whether content breaks a rule doesn’t tell you whether it was AI-generated or manipulated.
The question Your classifiers and hash matching ✦Resemble
Does this content break policy? Harm taxonomies, hash matching against known material, and your own trained classifiers. Nothing to add here, this part already works.
Was this generated by a model? Not assessed. A hash only matches material that has been seen before, and a new generation has never been seen before. Pixel and frame level detection across image, video and audio, tested against more than 250 generative models.
Was a real photo altered? Usually out of scope, since the content itself may not break any rule. Editing, splicing and tampering after capture, flagged separately from full generation.
Does it carry a provenance credential? Read where present, though most uploads arrive with the credential stripped. Checked, and detection still returns a verdict when no credential survives.
Can you defend the label or the takedown? A score and a policy reference, which is thin when a decision is appealed or reviewed externally. A report naming the artifacts behind the verdict, written at the moment of the decision.

What Resemble AI sends back

One API call returns the detection and the report together.
Result and risk level
Deepfake or authentic, with a confidence score and a risk level.
Abnormalities
The specific artifacts that triggered the flag, such as distorted features, lighting that does not match, over-smoothed skin, or gibberish text on a document.
Digital alteration
Editing, splicing, and tampering after capture, flagged even when noting was AI-generated, which is how most Photoshpped ID's get through.
Provenance and watermark
Whether a C2PA credential or a watermark survived, which matters as AI labeling obligations move from voluntary to expected.

5 questions to take to your next policy review

Most trust and safety teams can describe their harm taxonomy in detail and cannot say how much of their queue is synthetic. These five questions close that gap.
‍
What share of what you took down last month was AI generated, and how do you know?
When a user appeals a synthetic media decision, what evidence goes into the response?
How long can a check take at upload before your product team refuses to ship it?
What false positive rate would your moderators tolerate before they stop trusting the flag?
If a regulator asked how you decide to label AI content, could you show them the evidence behind a single decision?

Where it runs, and what happens to your data

Where it runs
Cloud, your own VPC, or fully on-prem. Air-gapped deployments are already in production with goveernment agencies.
What we keep
In memory processing is available, so a file is analyzed without being stored. Everything is encrypted at rest and in transit.
Certification
ISO/IEC 27001:2022 certified, and GDPR and HIPAA compliant. Full detail is in our trust center.
95.8%
Accuracy on images, across 1,000 test files
250+
Generative models tested against
0.12
Real-time factor, fast enough to run inline
Frequently asked questions
Does this classify harm categories such as CSAM, self harm or violence?
It does not, and that is deliberate. Resemble answers whether media was generated or altered, which is a different axis from whether it breaks policy. Teams run it alongside their harm classifiers rather than instead of them, and the two answers together are what make a decision defensible.
What is your false positive rate on images?
Around 4% on our published image benchmark, and lower on video. We would rather give you that number up front than have you find it in a pilot, because at moderation volume the false positive rate decides whether your team keeps trusting the flag.
Will it hold up on our content, not just a benchmark?
That depends on your pipeline, and compression and transcoding are the usual reason a model that tests well performs worse in production. We calibrate on a sample of your real assets before anyone commits to a number.
What happens when a new image or video generator comes out?
Coverage lags a new generator by a short window. We take early access to new models where we can get it, and new generations are folded into training as they appear, so detection on a generator that was weak at launch improves over later releases. The useful question is not the total model count but our current coverage for the generators turning up in your queue, and we will tell you what it is.
Can it run at upload without slowing the product down?
Yes, through the API at ingestion, and platforms running millions of assets a month do exactly that. Batch processing is available for backfills and for anything that does not need an answer immediately.
What if the file has no C2PA credential or watermark?
Detection still returns a verdict. Provenance credentials are useful when they survive, but most uploads arrive stripped of metadata, so a system that depends on them will miss almost everything that matters.
Can you detect AI generated text as well?
Yes, through a separate endpoint. It returns AI, human or uncertain, so a genuinely ambiguous passage gets an abstention rather than a forced call, and it needs at least 25 words to return anything. The accuracy figures on this page cover image, video and audio only, so text is a separate conversation with its own numbers.
Know what's real — and what's a real threat.
Join thousands of developers and enterprises detecting AI fraud and protecting their content with Resemble AI
Contact us