Imported from arcjet/well-known-bots (AGENTS.md). Install upstream with npx skills add arcjet/well-known-bots. Copyright stays with the author.

Agent Onboarding Guide

This document helps coding agents work efficiently with the well-known-bots repository.

Repository Overview

This repository maintains a curated list of well-known bots, crawlers, validators, monitors, and spiders in a single JSON file (well-known-bots.json). The file contains 600+ bot definitions with regex patterns for User-Agent matching and metadata for verification.

Key Files

• well-known-bots.json: Main data file (~13,653 lines) containing bot definitions
• validate.ts: Validation and formatting script - THE MOST IMPORTANT TOOL
• .github/workflows/ci-validation.yml: CI workflow that runs validation on every push/PR

Working with This Repository

Validation Dependencies

This is a data repository with no root package.json or project installation. The built-in validator and CIDR tests use only Node.js built-in modules. JSON Schema validation and strict type checking use the isolated tools/schema/ package. Use Node.js 26 to execute the TypeScript validators directly without a build step.

• Do not create a root package.json or install dependencies at the repository root.
• Install validation dependencies with npm ci --prefix tools/schema --ignore-scripts.
• Type-check both validators with npm run --prefix tools/schema typecheck.
• Run schema validation with node tools/schema/validate.ts.
• Keep validation dependency versions exact and commit the generated lockfile. For an intentional update, run npm install --prefix tools/schema --ignore-scripts --save-exact <package>@<version>, review the lockfile, and run npm audit --prefix tools/schema plus all validation checks.

Validation Script (Critical)

The validate.ts script is your primary tool. It has TWO modes:

1. Check mode: node validate.ts --check

   • Validates JSON formatting (2 spaces, proper newlines)
   • Validates all required fields exist and have correct types
   • Validates regex patterns compile correctly
   • Validates instances match/don't match their patterns
   • ALWAYS run this before committing any changes

2. Generate mode: node validate.ts --generate

   • Automatically reformats the JSON file with correct formatting
   • Use this if formatting is incorrect
   • IMPORTANT: Only use when you need to fix formatting

Making Changes to well-known-bots.json

When adding or modifying bot entries, refer to the README.md for:

• Complete field descriptions and structure
• Available categories
• Verification methods and examples
• How to add a new bot

Quick reference - Required fields for every bot entry:

• id: string - Unique identifier (kebab-case)
• categories: array - At least one category
• pattern: object with accepted and forbidden arrays of regex strings
• verification: array - Can be empty [] or contain verification methods

Pattern validation rules:

• All regex patterns in pattern.accepted must be valid
• All regex patterns in pattern.forbidden must be valid
• If instances.accepted is provided, all strings must match ALL accepted patterns and NONE of the forbidden patterns
• If instances.rejected is provided, strings must either not match all accepted patterns OR match at least one forbidden pattern

Common Workflows

Adding a New Bot

1. Edit well-known-bots.json to add your bot entry (see README.md for structure)
2. Validate your changes: node validate.ts --check
3. If formatting is wrong, auto-fix it: node validate.ts --generate
4. Validate again to ensure correctness: node validate.ts --check

Modifying an Existing Bot

1. Find the bot entry in well-known-bots.json
2. Make your changes
3. Always validate: node validate.ts --check

CI/CD Pipeline

The repository uses GitHub Actions for validation:

• Trigger: Runs on every push, pull request, and merge group
• What it does: Installs the locked schema dependencies with scripts disabled, then type-checks both validators and runs node validate.ts --check, node --test schema.test.js validate.test.cjs tools/schema/validate.test.cjs, and node tools/schema/validate.ts
• Node version: 26.x
• Location: .github/workflows/ci-validation.yml

All PRs must pass validation before merging.

Common Errors and Solutions

Error: "JSON file format is wrong"

Cause: Incorrect indentation or line breaks in the JSON file.

Solution:

node validate.ts --generate
node validate.ts --check

Error: "Item is missing required X field"

Cause: A bot entry is missing a required field.

Solution: Add the missing field. Required fields are: id, categories, pattern, verification.

Error: "Instance in instances.accepted does not match the required accepted pattern"

Cause: An example user-agent string in instances.accepted doesn't match the regex patterns in pattern.accepted.

Solution: Either fix the regex pattern or fix the example instance to match.

Error: "Pattern entry was not a string"

Cause: A pattern in pattern.accepted or pattern.forbidden is not a string.

Solution: Ensure all patterns are strings, not other types.

Error: "Invalid regex pattern"

Cause: A regex pattern cannot be compiled by JavaScript's RegExp.

Solution: Fix the regex syntax. Remember to escape special characters properly.

File Editing Best Practices

1. JSON Editing:

   • Use 2-space indentation
   • Keep the JSON structure consistent with existing entries
   • Let node validate.ts --generate handle formatting if unsure

2. Regex Patterns:

   • Test patterns before adding them
   • Escape special regex characters: \ (backslash for literal chars like \/)
   • Remember: Patterns are tested with JavaScript's RegExp engine

3. Verification Data:

   • Only add verification if the bot provider offers a reliable verification method
   • Most bots (570/600) have no verification - this is normal
   • Verify URLs are accessible before adding them. If you cannot load the URL then flag it for human verification in the pull request.

Testing Your Changes

Run the built-in validation, regression tests, and JSON Schema validation:

npm ci --prefix tools/schema --ignore-scripts
npm run --prefix tools/schema typecheck
node validate.ts --check
node --test schema.test.js validate.test.cjs tools/schema/validate.test.cjs
node tools/schema/validate.ts

Tips for Agents

1. Always validate before committing changes
2. Don't create build artifacts - this is a data-only repo
3. Read the README.md for context on the project's purpose
4. Check existing entries for examples when adding new bots
5. Use node validate.ts --generate to fix formatting issues automatically
6. Focus on data quality - this file is consumed by other projects

Example Bot Entry

A minimal valid bot entry:

{
  "id": "example-bot",
  "categories": ["search-engine"],
  "pattern": {
    "accepted": ["ExampleBot\\/"],
    "forbidden": []
  },
  "verification": []
}

For detailed examples with verification methods, see the Verification Methods section in the README.

Need Help?

• Check existing bot entries in well-known-bots.json for examples
• Review validate.ts to understand validation rules
• See the README.md for project background and usage