Overview
This project focuses on implementing a simple AI text detector, mirroring the functionality found in tools like Substack’s built-in AI detection feature. The goal is to provide a practical demonstration of how AI detectors operate, incorporating dataset construction, model training, local deployment, and an RLVR component. The project’s outcome includes an AI-detector API and a user interface for evaluating text generation.
Model Architecture
The core of the detector utilizes a DistilBERT classifier, a smaller, faster version of BERT, fine-tuned for this specific task. The classifier outputs a score between 0 and 100, representing the estimated probability that the input text was generated by an AI model. This score is derived from the classifier’s probability distribution for the AI-generated class, acknowledging that this score shouldn’t be interpreted as a general probability of AI authorship.
Detection Approach
The approach draws inspiration from methods used behind Substack’s AI detection feature, including techniques like Pangram models. These models analyze text for patterns indicative of AI generation, which can then be detected by the classifier. The project also references previous research on AI text detection, including supervised classifiers, perturbation-based probability tests, perplexity measures, and watermarking.
Deployment and Evaluation
The final project results in an AI-detector API and a user-friendly browser interface. This interface provides a whole-text AI score and highlights scores for individual text chunks. The system is designed for local deployment, allowing for immediate testing and experimentation. The project acknowledges the ongoing ‘cat-and-mouse’ game between AI detectors and generative models, highlighting the potential for false positives and the need for continuous model updates.
Source: https://magazine.sebastianraschka.com/p/ai-detector-from-scratch
