Blocking Harmful Content Uploads in Next.js AI Services

Any AI service that lets users upload files needs a proactive safeguard against publishing adult (harmful) content from day one. Once a file is published, it can spread quickly through caches and screenshots, so removing it after the fact has real limits.

This document lays out an 'Isolated Upload' architecture, step by step, that can be applied in a Next.js environment. The core idea is to control access so that no file is exposed to the outside by any route until verification is complete. The code examples are written around AWS S3 and the OpenAI Moderation API, but the same architecture works with other storage services or classification APIs.

mindmap
  root(("브라우저"))
    n1("격리버킷")
      n2("서버 검사")
        n3("공개버킷")
        n4("사람검토")
          n5("삭제")

Isolated Upload Setup

The isolated upload approach starts by splitting storage into two buckets. The quarantine bucket stays private with external access blocked, and only the public bucket is connected to the CDN. Users can only upload files to the quarantine bucket.

File transfers are set up so they don't pass through the Next.js server. The server generates a one-time upload URL (presigned URL) and hands it to the browser, which then uploads directly to storage. This keeps server load down and handles file size limits efficiently.

// app/api/upload-url/route.ts
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const s3 = new S3Client({ region: "ap-northeast-2" });

export async function POST(req: Request) {
  const user = await requireUser(req);          // 로그인 확인
  const { contentType, size } = await req.json();

  if (!ALLOWED_TYPES.includes(contentType)) return new Response(null, { status: 415 });
  if (size > MAX_BYTES) return new Response(null, { status: 413 });

  const key = `quarantine/${crypto.randomUUID()}`;
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({
      Bucket: process.env.QUARANTINE_BUCKET,
      Key: key,
      ContentType: contentType,
    }),
    { expiresIn: 60 }                           // 60초 뒤 만료
  );

  await db.upload.create({ data: { key, userId: user.id, status: "pending" } });
  return Response.json({ url, key });
}

Here are four key points to follow when implementing this:

First, for security, don't use user-provided input in file names. Generate random names on the server and manage them there.

Second, record files in the database with a 'pending' status, and show 'Under review' to users.

Third, keep the validity period of file upload URLs as short as possible.

Fourth, verify file types based on actual content, not extensions. Extensions and Content-Type can be tampered with, so don't trust them.

Server Processing

Checks must run on the server. Checks run in the browser can be bypassed simply by sending requests directly. When the upload finishes, the browser calls a completion API, and the server passes the file in the quarantine bucket to the classification API. To let the classification API read the file, the server creates a short-lived, read-only URL and hands it over.

// app/api/upload-complete/route.ts
import OpenAI from "openai";
import { GetObjectCommand } from "@aws-sdk/client-s3";

const openai = new OpenAI();

export async function POST(req: Request) {
  const user = await requireUser(req);
  const { key } = await req.json();

  const upload = await db.upload.findFirst({ where: { key, userId: user.id } });
  if (!upload || upload.status !== "pending") return new Response(null, { status: 404 });

  const readUrl = await getSignedUrl(
    s3,
    new GetObjectCommand({ Bucket: process.env.QUARANTINE_BUCKET, Key: key }),
    { expiresIn: 120 }
  );

  const res = await openai.moderations.create({
    model: "omni-moderation-latest",
    input: [{ type: "image_url", image_url: { url: readUrl } }],
  });

  const score = res.results[0].category_scores.sexual;
  const status = await decide(key, score);      // 3단계에서 설명
  return Response.json({ status });
}

Image processing can be done immediately within the request. Long-running work such as video processing is subject to serverless function execution time limits, so it should be designed to go into a queue and be handled by a separate worker.

Also, prepare for the chance that the browser leaves without calling the completion API. Files that stay in 'pending' status beyond a set time should be cleaned up periodically, or inspection can be triggered by upload events from storage, so no step in the process gets missed.

Score Handling

The classification API doesn't split content into a simple yes-or-no for adult material. Instead, it returns a score between 0 and 1, and two thresholds divide results into three bands.

const BLOCK = 0.8;   // 이 이상이면 자동 차단
const REVIEW = 0.3;  // 이 이상이면 사람 검토

async function decide(key: string, score: number) {
  if (score >= BLOCK) {
    await s3.send(new DeleteObjectCommand({ Bucket: QUARANTINE, Key: key }));
    return setStatus(key, "rejected", score);
  }
  if (score >= REVIEW) {
    return setStatus(key, "review", score);     // 격리 버킷에 그대로 둠
  }
  const publicKey = key.replace("quarantine/", "public/");
  await s3.send(new CopyObjectCommand({
    Bucket: PUBLIC,
    Key: publicKey,
    CopySource: `${QUARANTINE}/${key}`,
  }));
  await s3.send(new DeleteObjectCommand({ Bucket: QUARANTINE, Key: key }));
  return setStatus(key, "approved", score);
}

The values 0.8 and 0.3 are examples. You should collect samples similar to what might actually be uploaded to your service, run them, check how many slip through and how many get caught, and adjust accordingly. The most important rule is that if an error occurs, don't let the file through. If the classification API doesn't respond or returns an error, the file stays in pending and becomes a retry target. Letting checks be skipped during an outage can be dangerous.

Video, Prompts, and AI-Generated Output

Other paths are blocked by the same principle. For video, use ffmpeg to extract frames at 1–2 second intervals and check each one like an image. If even one frame crosses the blocking threshold, reject the whole video. Since the number of frames is the cost, set an upper limit on uploadable video length.

Text prompts go through a text check before the generation request is processed. You can pass the string to the same classification API. AI-generated output goes through steps 2 and 3 in exactly the same way before it is shown to users. Checking only inputs leaves gaps for workarounds, so output checking is actually the stronger line of defense.

Classification APIs and Costs

The most economical order is to start with the free OpenAI Moderation and move to a paid service when you need video or more detailed classification. The prices below were checked on October 7, 2026, and are in US dollars.

OpenAI is free but has call limits, and its general harmfulness classification doesn't let you finely separate levels of explicitness. Video also requires you to extract frames yourself and send them. With Sightengine, running several models on one image deducts a count for each model, and video is processed by default at one frame per second, so the quota runs out quickly. AWS's dedicated video API is not cost-effective compared with frame extraction. For a 60-second video, the video API costs $0.10, while sending 2 frames per second through the image API costs $0.12.

After-the-Fact Safeguards and Legal Obligations

Automated checks are not perfect, so it's good to have safeguards that catch content even after it passes. Put a report button on all content, and when reports accumulate, switch the content to private automatically and review it again. For account penalties, record the number of violations on the account, and restrict uploads or suspend the account when violations repeat. Keep judgment logs that record the score, time, and result. They serve as evidence when adjusting thresholds and when disputes arise. To block re-uploads, store hashes of rejected files so the same file is blocked immediately if it's uploaded again.

Extremely Important Matters

Child sexual exploitation material must be handled differently from general adult content. Suspected files should not go into the general review queue and should be blocked immediately. Storage, reporting, and deletion procedures carry legal obligations, so it's safest to get expert review before launch.

In Korea, businesses above a certain size are obligated to take technical measures such as filtering illegally filmed material. Whether your service falls under this depends on user count and revenue thresholds, so this is also an item to check before launch.