Imported from bedkillerspacex-boop/codex-skill-library (typescript-content-safety/SKILL.md). Install upstream with npx skills add bedkillerspacex-boop/codex-skill-library --skill typescript-content-safety. Copyright stays with the author.

TypeScript Content Safety

Scope And Authorization

• In scope: Building moderation for user-generated or model-generated content in products you operate, under product policy and legal guidance.
• Out of scope: Mass-scraping third-party sites to "moderate" them; doxxing; discriminatory enforcement outside written policy.
• Prefer least-privilege access to raw content queues; audit reviewer actions.
• Redact sensitive user content from engineering tickets when possible.
• Pair with code-quality-standards; route prompt injection to llm-prompt-injection.

When To Use

• Shipping UGC comments, uploads, or AI chat with risk categories.
• Combining blocklists, classifiers, and human review in Node services.
• Defining fail-closed vs fail-open for generation and publish paths.
• Measuring precision/recall and false positive impact on creators.
• Implementing age-gated or region-specific policy packs.

Do Not Use As Primary

Domain Focus

Actions

Workflow

1. Confirm policy and success criteria

1. Record categories, severity, regions, SLA for human review, authorization.
2. Success: severe categories blocked pre-publish; FP rate within budget.
3. Legal/policy sign-off on category definitions.

2. Inventory content surfaces

- Text posts, chat, filenames, images, audio
- Model outputs (pre-user delivery)
- Admin-generated content

3. Pipeline design

export type SafetyCategory =
  | "hate"
  | "sexual"
  | "self_harm"
  | "violence"
  | "scam";

export type SafetyResult = {
  action: "allow" | "shadow" | "quarantine" | "block";
  categories: Partial<Record<SafetyCategory, number>>;
  model: string;
  policyVersion: string;
};

export async function moderateText(
  text: string,
  classify: (t: string) => Promise<Partial<Record<SafetyCategory, number>>>,
  thresholds: Record<SafetyCategory, number>,
): Promise<SafetyResult> {
  // Stage 0: size/empty
  if (!text.trim()) {
    return { action: "allow", categories: {}, model: "noop", policyVersion: "v1" };
  }
  // Stage 1: optional blocklist (high precision phrases)  -  keep small
  const scores = await classify(text);
  let action: SafetyResult["action"] = "allow";
  for (const [cat, score] of Object.entries(scores) as [SafetyCategory, number][]) {
    if (score >= thresholds[cat]) action = "block";
    else if (score >= thresholds[cat] * 0.8 && action === "allow") action = "quarantine";
  }
  return { action, categories: scores, model: "classifier", policyVersion: "v1" };
}

4. Enforce at write path

1. User publish: block/quarantine before durable write.
2. LLM output: filter before stream finalize; document fail behavior if API down.
3. Async: re-score on model upgrade; do not silently unban without review.

5. Human review and appeals

1. Reviewer UI with limited context; dual control for account bans.
2. Appeal path with SLA; audit who overrode model.
3. Training set hygiene: no leaking private UGC into public fine-tunes without rights.

6. Verify

pnpm vitest run tests/content-safety
# Fixture packs: true_positive.jsonl / true_negative.jsonl

7. Observe and hand off

1. Metrics: block rate, FP appeals, latency, outage fallback count.
2. Weekly policy sampling.
3. Route jailbreak/prompt attacks to LLM skills.

Good / Bad

Output Checklist

• Policy categories and thresholds documented
• Surfaces inventoried
• Pipeline + enforcement points implemented
• Human review/appeal path defined
• Fixture evaluation recorded
• Outage behavior documented
• Metrics live
• code-quality-standards applied
• Residual risk and review date noted

Rules

• Follow written product policy and law; eng does not invent speech codes alone.
• Protect reviewer mental health and data access least privilege.
• Never use moderation systems for unauthorized surveillance.
• Prefer measurable precision/recall over vibes.
• LLM security exploits route to dedicated skills.