Imported from wyattowalsh/proxywhirl (
AGENTS.md). Install upstream withnpx skills add wyattowalsh/proxywhirl. Copyright stays with the author.
AGENTS.md
AI agent instructions for proxywhirl — Python proxy rotation library. Human docs: README.md
Overview
| Property | Value |
|---|---|
| Language | Python 3.10+ (uv package manager) |
| Stack | pytest, ruff, ty (Astral), pre-commit, GitHub Actions |
| Docs | Fumadocs + Next.js in web/content/docs/; canonical prose in docs/; generated assets in docs/proxy-lists/ and docs/assets/ |
| Key Libs | httpx, pydantic, sqlmodel, loguru, tenacity, typer, fastapi |
Commands
| Task | Command |
|---|---|
| Install | uv sync |
| Test | just test-fast / uv run pytest tests/unit/ tests/integration/ tests/property/ -m "not slow and not stress" --timeout=120 -q |
| Lint | just lint / uv run ruff check proxywhirl/ tests/ |
| Format | just format / uv run ruff format proxywhirl/ tests/ |
| Type check | just type-check / uv run ty check proxywhirl/ |
| Quality gates | just quality-gates |
| Docs generate | pnpm --dir web run docs:generate |
| Docs build | just docs-html / pnpm --dir web run build |
| Source gate | uv run proxywhirl sources --validate --fail-on-unhealthy --timeout 5 --concurrency 5 |
Always prefix with uv run — never bare pytest, python, or pip.
Use uvx only for intentionally isolated tools such as uvx pip-audit or uvx bandit.
Parallelization
| Level | Tasks | Dependencies |
|---|---|---|
| 0 | lint, format-check, type-check | None |
| 1 | test-unit | Level 0 |
| 2 | test-integration, test-property | test-unit |
| 3 | coverage, build | All tests |
Locks required: uv.lock (write)
Key Modules
| Module | Purpose |
|---|---|
models/ |
Pydantic models: Proxy, ProxyPool, Session, *Config |
strategies/ |
RoundRobinStrategy, WeightedStrategy, PerformanceBasedStrategy |
rotator/ |
ProxyWhirl (sync), AsyncProxyWhirl (async) |
storage.py |
SQLiteStorage, FileStorage, ProxyTable (SQLModel) |
fetchers.py |
ProxyFetcher, ProxyValidator, parsers (JSON/CSV/PlainText/HTMLTable) |
sources.py |
ALL_SOURCES, RECOMMENDED_SOURCES, predefined proxy endpoints |
cache/ |
CacheManager, multi-tier caching |
circuit_breaker/ |
CircuitBreaker, AsyncCircuitBreaker |
retry/ |
RetryExecutor, RetryPolicy, backoff logic |
rate_limiting/ |
RateLimiter (token bucket) |
api/ |
FastAPI canonical /api/* routes |
mcp/ |
MCP server for AI assistants |
cli.py |
Typer CLI (proxywhirl command) |
browser.py |
Playwright for JS-rendered sources |
config.py |
TOML config, CLIConfig, DataStorageConfig |
exceptions.py |
ProxyWhirlError hierarchy |
safe_regex.py |
ReDoS-safe regex utilities |
geo.py |
IP geolocation enrichment |
utils.py |
Utility functions (encryption, URL parsing, logging config) |
logging_config.py |
Loguru logging configuration |
metrics_collector.py |
Prometheus metrics collection |
enrichment.py |
Proxy data enrichment |
premium_sources.py |
Premium proxy source definitions |
migrations.py |
SQLModel database migrations (Alembic) |
exports.py |
Data export utilities |
rwlock.py |
Read-write lock utilities |
Structure: proxywhirl/ (core), tests/ (unit/integration/property/benchmarks/contract), web/content/docs/ (active Fumadocs docs), docs/ (legacy/generated assets), examples/
Key Exports (from proxywhirl/__init__.py):
- Rotators:
ProxyWhirl,AsyncProxyWhirl - Models:
Proxy,ProxyPool,Session,ProxySource,HealthStatus,ProxyChain,SelectionContext,ProxyCredentials,ProxySourceConfig,SourceStats,HealthMonitor,ProxyFormat,RenderMode,ValidationLevel - Config:
ProxyConfiguration,StrategyConfig,CircuitBreakerConfig,RetryPolicy,CacheConfig,DataStorageConfig - Components:
CacheManager,CircuitBreaker,AsyncCircuitBreaker,RetryExecutor,BrowserRenderer,ProxyFetcher,ProxyValidator - Strategies:
RotationStrategy(protocol),StrategyRegistry, all 9 strategy classes - Parsers:
JSONParser,CSVParser,PlainTextParser,HTMLTableParser - Sources:
ALL_SOURCES,RECOMMENDED_SOURCES,ALL_HTTP_SOURCES,ALL_SOCKS4_SOURCES,ALL_SOCKS5_SOURCES,API_SOURCES - Exceptions:
ProxyWhirlError,ProxyPoolEmptyError,ProxyValidationError,ProxyConnectionError,ProxyAuthenticationError,ProxyFetchError,ProxyStorageError,RetryableError,NonRetryableError,RegexTimeoutError,RegexComplexityError,CacheCorruptionError,CacheStorageError,CacheValidationError,RequestQueueFullError - Utils:
configure_logging,encrypt_credentials,decrypt_credentials,generate_encryption_key,deduplicate_proxies,is_valid_proxy_url,parse_proxy_url,validate_proxy_model,proxy_to_dict,create_proxy_from_url
Testing
pytest with asyncio_mode=auto. Markers: slow, integration, unit, benchmark, snapshot, property, flaky, network, stress. Fixtures in tests/conftest.py.
Use the same tiers as CI:
| Scope | Command | Notes |
|---|---|---|
| Focused regression | uv run pytest tests/unit/test_<module>.py -q --no-cov -n0 |
Keeps local probes free of coverage/xdist noise. |
| CI parity | just quality-gates |
Runs lint, type check, non-slow unit/integration tests, and coverage. |
| Expanded non-slow | uv run pytest tests/ -q -m "not slow" --ignore=tests/benchmarks --timeout=120 |
Covers API, contract, property, unit, and integration surfaces without real-network slow/browser tests or benchmarks. |
| Full raw suite | uv run pytest tests/ -q |
Intentional stress run only; includes slow real-browser/network tests and benchmarks that are not the CI parity gate. |
When manually testing CLI fixes, use uv run proxywhirl --no-lock --help plus a temporary config path and cover pool add/list/remove, config set/get, import-proxies, validate-proxy, strict sources --validate, and credential redaction. When API/runtime behavior changes, start uv run uvicorn proxywhirl.api:app --host 127.0.0.1 --port <port> with a temporary PROXYWHIRL_STORAGE_PATH and verify /api/health plus /api/ready.
Code Style
- Naming:
PascalCase(classes),snake_case(functions),UPPER_SNAKE_CASE(constants),_prefix(private) - Ruff: 100 char lines, double quotes, 4-space indent. Rules: E, F, I, N, W, UP, B, C4, SIM
- Types: Always type-hint public functions. Use
from __future__ import annotations. Pydantic withConfigDict(frozen=True, extra="forbid")
Git
Branches: main (protected), develop, feature/*, fix/*, claude/*
Commits: <type>(<scope>): <description> — types: feat, fix, docs, chore, refactor, test, perf
Pre-commit: ruff lint/format, commitizen, private key detection, large file check
CI Workflows (.github/workflows/): ci.yml (main), security.yml, generate-proxies.yml (3h cron), validate-sources.yml, update-readme-stats.yml, release.yml, cd.yml
For remote assurance, monitor at least CI, Security Scan, and manually dispatched Validate Proxy Sources on the pushed SHA. Treat GitHub log markers as actionable if they include ##[warning], ##[error], ::warning::, ::error::, Python warning classes, tracebacks, or Codecov upload token errors. Literal environment assignments such as PYTHONWARNINGS="ignore::UserWarning" are not emitted warnings.
Codecov upload in ci.yml is optional and gated by CODECOV_TOKEN_PRESENT; coverage threshold enforcement and coverage artifact upload must continue even when Codecov is skipped.
Documentation
The active docs framework is Fumadocs on Next.js in web/.
- Source pages live in
web/content/docs/**/*.mdx(guides, concepts, interfaces, reference, operations). - Next.js route and UI source lives in
web/src/app/,web/src/components/, andweb/src/screens/. - Generated reference pages live in
web/content/docs/generated/and are regenerated byweb/scripts/generate-docs.mjs. - Proxy-list web assets are copied from
docs/proxy-lists/intoweb/public/proxy-lists/bypnpm --dir web run docs:generate. - After docs, README, AGENTS, public API surface, file-structure, agent, or skill-definition changes, invoke the docs-steward workflow and run the relevant docs checks.
- Use
shieldcnbadges inREADME.md; keep the top badge row to 3–6 clickable SVG badges and do not leave placeholder owner/repo/package values. - For rendered web UI, verify internal navigation uses Next
Link, icon-only controls have accessible names, decorative icons/images are hidden from assistive tech, broadtransition-allis avoided, reduced-motion preferences are honored, and no emoji are used as UI icons.
Docs validation sequence:
pnpm --dir web run docs:generate
pnpm --dir web run lint
pnpm --dir web run test:run
pnpm --dir web run build
Run pnpm --dir web run test:e2e only when browser dependencies are installed and the change affects rendered docs behavior.
Boundaries
Always:
- Run
just lintbefore commit - Run
just quality-gatesbefore production-readiness commits when Python behavior changed - Add tests for new features (test-first)
- Use type hints on public functions
- Use
logurufor logging (neverprint) - Prefix all Python commands with
uv run - Validate inputs at module boundaries
- Keep docs generated from source: edit generator scripts or source docs, then run
pnpm --dir web run docs:generate
Ask First:
- New dependencies in
pyproject.toml - Schema changes (SQLModel models)
- Public API changes (
__init__.py) - CLI command changes
- CI/CD workflow modifications
- Changes affecting >5 files
Never Touch:
.envfiles (secrets)uv.lock(auto-generated).venv/directorydist/,*.egg-info/*.backup,*.bakfiles- Files outside repo root
Database: proxywhirl.db is local runtime state (gitignored). CI refreshes exported proxy lists under docs/proxy-lists/ every 3h.
Environment Variables
| Variable | Component | Purpose |
|---|---|---|
PROXYWHIRL_KEY |
CLI/config | Master encryption key |
PROXYWHIRL_CACHE_ENCRYPTION_KEY |
cache | Fernet key for L2 cache |
PROXYWHIRL_STORAGE_PATH |
api | SQLite database path |
PROXYWHIRL_API_KEY |
api | API authentication key |
PROXYWHIRL_MCP_API_KEY |
mcp | MCP server auth key |
PROXYWHIRL_MCP_ALLOW_UNAUTHENTICATED_WRITES |
mcp | Local-dev/test override for MCP write actions |
PROXYWHIRL_MCP_DB |
mcp | MCP database path |
See subsystem AGENTS.md files for complete environment variable lists.
Security
| Category | Forbidden | Allowed |
|---|---|---|
| Files | *.env*, *.pem, *.key, *secret*, *credential*, *.bak |
proxywhirl/, tests/, docs/, examples/ |
| Commands | rm -rf, curl | sh, eval, chmod 777, sudo * |
uv run *, just *, git * |
| Network | Raw sockets, production APIs, external URLs not in codebase | httpx client, localhost, mocked fixtures |
| Secrets | Hardcoded creds, logging passwords, base64-encoded keys | os.environ.get(), cryptography lib |
| Regex | User-provided patterns without validation | safe_regex.py utilities only (ReDoS) |
| Destructive | Bulk deletes, schema drops, force pushes | Single-file ops, migrations, regular commits |
Human Intervention Required
- Modifying
__init__.pyexports - Database schema migrations
- CI/CD workflow changes
- New external dependencies
- Any operation on
mainbranch
Gotchas
| Issue | Fix |
|---|---|
ModuleNotFoundError |
uv sync |
| Async test errors | asyncio_mode = "auto" in pyproject.toml |
| Type errors | Use uv run ty check not mypy |
| Import errors | Use uv run pytest not bare pytest |
| Raw full suite hangs | Use CI parity gates; slow browser/benchmark tests are intentional stress surfaces |
| HTTP mock not working | Use respx not responses; check respx_mock fixture |
| Pydantic validation | Use ConfigDict(extra="forbid") to catch typos |
| Credential exposure | Use SecretStr, check redact_url() in exceptions.py |
| MCP tests skip | MCP requires Python 3.10+; tests auto-skip on older |
Setup
git clone https://github.com/wyattowalsh/proxywhirl.git && cd proxywhirl
uv sync && pre-commit install
just --list # see all available developer commands
Subagents
Scope: proxywhirl/**/*.py, tests/**/*.py — Forbidden: .env, uv.lock, .venv/, dist/
Context template:
Working dir: /path/to/proxywhirl
Package manager: uv (ALWAYS `uv run` prefix)
Test command: uv run pytest tests/unit/test_<module>.py -v
Lint command: uv run ruff check proxywhirl/<module>.py
Docs command: pnpm --dir web run docs:generate && pnpm --dir web run build
Decision Tree
| Scenario | Action |
|---|---|
| Sync vs Async | ProxyWhirl for scripts/CLI; AsyncProxyWhirl for web/high-concurrency |
| Tests fail | 1) uv run pytest path::test -v 2) Check conftest.py 3) uv sync |
| Lint fails | 1) just format auto-fix 2) Manual fix remaining 3) Check ruff codes |
| Type errors | uv run ty check (not mypy) — check ty.rules in pyproject.toml |
| Docs drift | 1) pnpm --dir web run docs:generate 2) Inspect generated diff 3) pnpm --dir web run build |
| Source drift | uv run proxywhirl sources --validate --fail-on-unhealthy --timeout 5 --concurrency 5 |
| New strategy | Implement RotationStrategy protocol, register in StrategyRegistry |
| New exception | Inherit from ProxyWhirlError, add error code to ProxyErrorCode |
| HTTP mocking | Use respx (not responses), fixtures in conftest.py |
