Imported from clolckliang/aigc-detector-toolkit (skills/aigc-detect/SKILL.md). Install upstream with npx skills add clolckliang/aigc-detector-toolkit --skill aigc-detect. Copyright stays with the author.

AIGC Detector Toolkit — Skill Reference

Chinese AI-generated text (AIGC) detection and refinement toolkit with five engines, model training, data generation, evaluation, and a one-click 降AIGC率 refinement workflow.

Project Root

All commands assume the working directory is the toolkit root:

/home/shaowenliang/Desktop/aigc-detector-toolkit/

Architecture Overview

┌─────────────────────────────────────────────────────┐
│                    main.py (CLI)                     │
│  detect | batch | refine | eval | train | status | test │
├─────────────────────────────────────────────────────┤
│                  core/engine.py                      │
│         DetectionEngine (5-engine fusion)            │
│  ┌──────────┬────────┬────────┬─────────┬─────────┐ │
│  │ FengCi0  │  HC3   │ OpenAI │Bino-    │ Local   │ │
│  │ (RF)     │ (CNN)  │ (API)  │culars   │Logprob  │ │
│  │ 12-dim   │ m3e    │ logprobs│(API)   │(opt-in) │ │
│  │ features │ 768-dim│ ppl    │ppl/x_ppl│LogRank  │ │
│  └──────────┴────────┴────────┴─────────┴─────────┘ │
├──────────────────┬──────────────────────────────────┤
│  core/refiner.py │  core/trainer.py                 │
│  降AIGC率工作流   │  FengCiTrainer / HC3Trainer      │
│  3策略:           │  数据增强 / 阈值校准              │
│  refine_cn        │  generate_sample_data.py         │
│  deai_cn          │                                  │
│  deai_en          │                                  │
├──────────────────┴──────────────────────────────────┤
│  extractors/          │  reporters/                  │
│  docx / md / txt      │  console / txt / json / html │
├─────────────────────────────────────────────────────┤
│  models/  (pretrained) │  configs/  │  examples/    │
│  fengci/ (1MB RF)      │  default   │  eval_sample  │
│  hc3/    (25MB CNN)    │  .yaml     │  .jsonl       │
└─────────────────────────────────────────────────────┘

CLI Commands

detect — 单文件 AIGC 检测

uv run python main.py detect <file> [--engine ENGINE] [--threshold FLOAT] [-o DIR]

• Supported formats: .docx, .md, .txt
• Auto-extracts paragraphs, skips headings/captions/short text
• Generates: AIGC检测报告_*.txt + AIGC检测结果_*.json + AIGC可视化报告_*.html

batch — 批量检测目录

uv run python main.py batch <directory> [--engine ENGINE] [--threshold FLOAT] [-o DIR]

• Recursively finds all .docx/.md/.txt files
• Generates per-file reports + summary

refine — 降AIGC率工作流（核心差异化功能）

uv run python main.py refine <file> [--threshold FLOAT] [--no-recheck] [-o DIR]

Requires OPENAI_API_KEY for LLM-based text refinement.

Workflow:

1. Extract paragraphs from document
2. Run AIGC detection (5-engine fusion) on every paragraph
3. Classify high-risk paragraphs (score > threshold, default 0.4)
4. Auto-select refinement strategy per paragraph:
   • refine_cn — Chinese expression polishing (fix grammar, remove colloquialisms)
   • deai_cn — Chinese AI-style removal (eliminate AI clichés, add personal touches)
   • deai_en — English AI-style removal (natural academic rewriting)
5. Re-detect refined paragraphs (unless --no-recheck)
6. Generate: 降AIGC报告_*.txt + 润色后文本_*.docx + 润色结果_*.json

eval — 评测标注数据集

uv run python main.py eval <dataset.jsonl> [--engine ENGINE] [--threshold FLOAT] [-o DIR]

Dataset format (JSONL, one per line):

{"label": "human", "text": "..."}
{"label": "ai", "text": "..."}

Outputs: accuracy, precision, recall, F1, AUC, confusion matrix, threshold scan.

train — 训练/微调检测模型

# FengCi0 Random Forest
uv run python train.py --engine fengci --data-dir data/train/

# HC3 CNN
uv run python train.py --engine hc3 --data-dir data/train/ --epochs 30

# With data augmentation
uv run python train.py --engine fengci --data-dir data/train/ --augment

# Generate AI training data via LLM
uv run python scripts/generate_sample_data.py --count 200

Training data structure:

data/train/
├── ai/          # AI-generated texts, one .txt per sample
│   ├── 0001.txt
│   └── ...
└── human/       # Human-written texts, one .txt per sample
    ├── 0001.txt
    └── ...

status — 引擎状态

uv run python main.py status

test — 快速自测

uv run python main.py test

Five Engines — Deep Dive

1. FengCi0 (Feature Engineering + Random Forest)

Core file: core/fengci_adapter.py

12-dimensional text features (all self-contained, no external repo needed):

Algorithm: DC resistance — human text has higher entropy, more diverse vocabulary, more varied sentence lengths.

Model: sklearn RandomForestClassifier (default), with LogReg and GradientBoosting as candidates. Best selected automatically.

2. HC3 + m3e (Deep Learning)

Core file: core/hc3_adapter.py

• Encoder: moka-ai/m3e-base SentenceTransformer (768-dim)
• Classifier: Custom OptimizedCNN (Conv1d → BN → Pool → Conv1d → BN → Pool → FC → FC)
• Training data: HC3-Chinese dataset (Hello-SimpleAI/HC3-Chinese)

3. OpenAI-Compatible API (Logprobs)

Core file: core/openai_adapter.py

Uses any OpenAI-compatible API's logprobs feature to compute text perplexity.

Strategy: perplexity (default) — splits text into prefix/suffix, asks model to continue from prefix, measures surprise of actual suffix.

Fallback: If API lacks logprobs, degrades to basic text statistics (sentence CV, avg length).

4. Binoculars (ICLR 2024)

Core file: core/binoculars_adapter.py

Based on paper: "Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text"

Formula: score = perplexity / cross_perplexity

• observer mode: "Please reproduce this text verbatim"
• performer mode: "Please rewrite this text in your own words"

Lower ratio → more likely AI-generated.

5. Local Logprob (Opt-in)

Core file: core/local_logprob_adapter.py

Loads a local HuggingFace causal LM and computes LogRank/perplexity directly.

Requires transformers and a local model path. Disabled by default (weight=0).

Refinement Strategies (降AIGC率)

refine_cn — 表达润色

• Target: AIGC-scored paragraphs with moderate AI probability
• Philosophy: 克制修改 — minimal intervention
• Actions: fix grammar, remove colloquialisms, preserve author's style
• Prompt: "如果你原文表达已经清晰、准确且符合学术规范，请务必保留原样"

deai_cn — 去AI味（中文）

• Target: Paragraphs with 2+ AI cliché markers (综上所述, 值得注意的是, 从本质上讲...)
• Actions:
  • Eliminate template phrases
  • Diversify sentence structures
  • Add personal markers (本文认为, 实验发现, 在调试过程中)
  • Replace vague generalizations with specific technical details

deai_en — 去AI味（英文）

• Target: English paragraphs
• Actions:
  • Replace overused words (leverage → use, delve into → investigate)
  • Remove mechanical transitions (First and foremost → direct connection)
  • Convert bullet lists to flowing prose
  • Vary sentence length

Paragraph Classification Logic

def classify_paragraph(text):
    if len(text) < 30:          return 'skip'
    if is_code_or_formula:       return 'skip'
    if is_reference:             return 'skip'
    if is_caption:               return 'skip'
    if is_english:               return 'deai_en'
    if ai_marker_count >= 2:     return 'deai_cn'
    return 'refine_cn'

Configuration Reference

configs/default.yaml

engine:
  default: "ensemble"
  ensemble:
    fengci_weight: 0.30
    hc3_weight: 0.20
    openai_weight: 0.25
    binoculars_weight: 0.25
    local_logprob_weight: 0.0

openai_api:
  api_base: "https://api.openai.com/v1"
  api_key: ""
  model: "gpt-4o-mini"
  strategy: "perplexity"

binoculars:
  api_base: ""    # empty → reuse openai_api
  api_key: ""
  model: ""

local_logprob:
  model_name_or_path: ""  # empty → disabled
  device: "cpu"
  max_length: 512
  stride: 256
  method: "logrank"

extraction:
  min_paragraph_length: 30
  skip_headings: true
  skip_captions: true
  skip_code_blocks: true

threshold:
  aigc_threshold: 0.5

performance:
  api_concurrency: 1

output:
  dir: null    # null → same dir as input file
  json: true
  text: true
  html: true
  console: true
  verbose: false

Environment Variables

Python API

Detection

from core.engine import DetectionEngine

engine = DetectionEngine(mode="ensemble")  # or "fengci", "hc3", "openai", etc.

# Single paragraph
result = engine.detect_single("待检测文本...")
print(result["aigc_score"])     # 0.0 - 1.0
print(result["label"])          # "ai" or "human"
print(result["confidence"])     # 0.0 - 1.0
print(result["engine_results"]) # per-engine breakdown

# Batch
results = engine.detect_batch(["text1", "text2", "text3"])

Refinement

from core.refiner import RefinementEngine, generate_refinement_report, export_refined_docx

refiner = RefinementEngine(
    api_base="https://api.openai.com/v1",
    api_key="sk-...",
    model="gpt-4o-mini",
)

# Single paragraph
refined, note = refiner.refine_paragraph("综上所述，...", strategy="deai_cn")

# Batch (with detection results from DetectionEngine)
ref_results = refiner.refine_batch(paragraphs, detection_results, score_threshold=0.4)

# Generate reports
generate_refinement_report(ref_results, "报告.txt")
export_refined_docx(paragraphs, ref_results, "润色后.docx")

Evaluation

from core.evaluation import load_labeled_dataset, summarize_engine_metrics, write_eval_report

dataset = load_labeled_dataset("eval.jsonl")
texts = [d["text"] for d in dataset]
results = engine.detect_batch(texts)
summary = summarize_engine_metrics(dataset, results, threshold=0.5)
write_eval_report(summary, "eval_report.json")

Training

from core.trainer import FengCiTrainer, HC3Trainer

# FengCi0
trainer = FengCiTrainer(output_dir="models/fengci")
metrics = trainer.train(texts, labels, augment=True)
print(metrics["best_model"], metrics["test_metrics"])

# HC3
trainer = HC3Trainer(output_dir="models/hc3")
metrics = trainer.train(texts, labels, epochs=30)

Score Interpretation

Workflow: Improve Detection Accuracy

1. Generate AI data
   uv run python scripts/generate_sample_data.py --count 500

2. Collect human texts → data/train/human/

3. Retrain
   uv run python train.py --engine fengci --data-dir data/train/

4. Evaluate
   uv run python main.py eval examples/eval_sample.jsonl

5. Tune threshold in configs/default.yaml

Workflow: Reduce AIGC Score (降AIGC率)

1. Detect current state
   uv run python main.py detect 论文.docx

2. Set API key for refinement
   export OPENAI_API_KEY="sk-..."

3. One-click refinement
   uv run python main.py refine 论文.docx --threshold 0.3

4. Check results
   - 降AIGC报告_*.txt — before/after comparison
   - 润色后文本_*.docx — refined document
   - Re-detect refined doc to verify improvement

Extending the Toolkit

Add a New Detection Engine

1. Create core/your_engine_adapter.py with a class implementing:

   class YourEngineDetector:
       available: bool
       def detect(self, text: str) -> Dict  # returns {"aigc_score": float, "label": str, ...}
       def detect_batch(self, texts: List[str]) -> List[Dict]

2. Register in core/engine.py:

   • Add import
   • Add to ENGINE_NAMES
   • Add constructor parameter
   • Add initialization block

3. Add CLI argument in main.py:

   • Add to choices list
   • Add to build_engine() parameter passing

4. Add config section in configs/default.yaml

Add a New Refinement Strategy

1. Define prompt template in core/refiner.py:

   PROMPT_YOUR_STRATEGY = """Your prompt here... {paragraph}"""

2. Add to classify_paragraph() logic

3. Add to refine_paragraph() strategy dispatch

Key Files Reference

Known Limitations & Notes

• HC3 engine may fail with certain transformers versions due to is_torch_npu_available import error. FengCi0 + OpenAI API + Binoculars still work.
• FengCi0 uses jieba for Chinese word segmentation. Ensure jieba is installed.
• Refinement requires an active OpenAI-compatible API with OPENAI_API_KEY set.
• Local Logprob engine is opt-in and disabled by default (weight=0). Set local_logprob_weight > 0 and provide a model path to enable.
• Binoculars API mode requires API key. Without it, the engine is skipped.
• Score interpretation varies by document type. Academic papers naturally have lower AIGC scores than product descriptions.