Moderating user-generated AI images

Decide what you are actually moderating for

Teams often start by shopping for a classifier and never write down the policy it is supposed to enforce. Do it the other way around. Your policy is driven by four things, and they rarely point the same way.

• The law where your users are — which varies enormously and changes faster than your roadmap.
• Your app store, if you have one. Mobile store rules are usually stricter than the law and are enforced by removal.
• Your payment processor. This is the one teams discover last and it is frequently the binding constraint — read your processor’s prohibited-business list before you build, not after your first hold.
• Your own brand. Perfectly legal output can still be something you do not want on your front page.

Three places to put a gate

Each position catches a different failure and costs a different amount. Most products end up with two of the three.

Before submit (prompt)

Cheapest — rejects before you spend a credit. Misses anything phrased innocuously.

On output (image)

The only check that sees what was actually produced. Costs one classifier call per image.

Before publish (share, feed, export)

Catches what the first two missed and scales with exposure, not generation.

Gate at the webhook, not in the request path

Generation is asynchronous here: you submit, you get a job id, and a signed webhook fires when the output is ready. That callback is the natural place to run your classifier, because it is off the user’s critical path and you are holding the URL before anyone else has seen it.

Verify the signature first — an unauthenticated handler that trusts its body is a way to get arbitrary URLs into your feed.

import { verifyFastgenSignature } from "@/lib/fastgen";

export async function POST(req: Request) {
  const raw = await req.text();
  if (!verifyFastgenSignature(raw, req.headers)) {
    return new Response("bad signature", { status: 401 });
  }

  const event = JSON.parse(raw);
  if (event.status !== "succeeded") return new Response("ok");

  // Your policy, your classifier. Hold the asset until it returns.
  const verdict = await classify(event.output[0].url);

  await db.generation.update({
    where: { jobId: event.id },
    data: {
      state: verdict.allowed ? "ready" : "withheld",
      labels: verdict.labels,
      reviewedAt: new Date(),
    },
  });

  return new Response("ok");
}

Copy anything you intend to keep

Output URLs expire after 7 days. That is a retention feature, not a CDN — if your product shows a user their history, copy the bytes into your own storage inside that window. Do the copy after your classifier passes, so a withheld asset never lands in your bucket in the first place.

Keep records you can actually answer a report with

The day someone reports an image, you need to answer three questions quickly: who generated it, what did they send, and what did you do about it. Store enough to answer them and no more than you are willing to be responsible for.

• The job id and your internal user id, so a reported asset resolves to an account.
• The prompt as submitted, and the classifier verdict with its labels and timestamp.
• The action you took and who took it — withheld automatically, or cleared by a named reviewer.
• A takedown path a stranger can find and use without an account. A contact address on a public page is the minimum.
• A retention window for all of the above, written down and actually enforced by a job.

What the platform does and does not do for you

Worth being unambiguous, because assuming a safety net that is not there is the expensive mistake. The platform blocks one thing at the prompt and passes everything else through, so any policy narrower than the legal floor is yours to implement.

Prompt filtering for adult content

Not performed. Yours.

Output nudity or violence detection

Not performed. Yours.

Minor-safety prompt gate

Enforced by the platform, always on.

Age verification of your end users

Yours.

Takedown handling for your users’ content

Yours.

Output retention

Deleted after 7 days by the platform.