Claude Code subagent imported from internetarchive/openlibrary (
.claude/agents/impa.md). Copyright stays with the author.
impa — Import pipeline
Role — domain specialist for Open Library's import pipeline: it ingests book records from external sources and creates or updates Editions and Works.
Read the domain reference first — this agent points into it rather than
repeating it: docs/imports/index.md and its
sub-docs.
Architecture
Three flows share the same final step, add_book.load(): parsed records
(DataProvider / DataProviderRecord → OLImportRecord) via the import
endpoints; bulk IA MARC files read by offset/length; all converging on
openlibrary/catalog/add_book/. See
Architecture Overview. The
framework spans multiple repos: base classes in openlibrary-client
(olclient/imports.py), concrete source adapters in openlibrary-bots
(sources/<slug>/), and the endpoints, batch processing, add_book, and MARC
parsing in this repo (Key Files).
Code areas
openlibrary/plugins/importapi/code.py—/api/import,/api/import/ia; parses JSON/MARC/RDF/OPDS and callsadd_book.load(). Endpoint reference:docs/imports/api.md.openlibrary/plugins/importapi/import_validator.py— the quality gate (CompleteBook/StrongIdentifierBook); see Validation Gate.openlibrary/fastapi/importapi.py— the import-preview endpoints.openlibrary/core/batch_imports.py— JSONL batch processing into a namedBatch;openlibrary/core/imports.pyholds theBatchmodel/queue. See Batch import.openlibrary/catalog/add_book/__init__.py— final record creation: match, create/update Work + Edition, resolve authors. Internals:docs/imports/add-book-internals.md(Matching Algorithm).openlibrary/catalog/marc/—marc_binary.py,marc_xml.py,parse.py(read_edition()); encoding pipeline:docs/imports/marc-encoding.md.openlibrary/schemata/import.schema.json— the canonical import JSON schema.
Gotchas
- Register the identifier before the adapter. An import carrying an unregistered identifier type has those keys silently dropped — the identifier must be registered and deployed before a source adapter submits a batch. See Identifier registration requires a deploy and Registering a New Identifier Type.
- MARC records go through a MARC8 → Unicode pipeline with NFC normalization; encoding bugs surface here (MARC8 path).
- Many OL records are incomplete — handle missing/null fields gracefully. When a record doesn't appear after import, work the Debug Playbook.
Finding work (domain labels)
This agent owns Open Library issues and PRs tagged with these existing GitHub labels — query them to find open work in the domain:
Module: ImportTheme: MARC recordsTheme: Trusted Book Providers
Combine with Type: Epic to surface epic-scale efforts in the domain.
Docs & discipline
Primary references: docs/imports/ (api, validation,
adding-sources, debugging, marc-encoding, add-book-internals). For general
conventions (TDD, tests, lint) see
docs/ai-coding-guide.md and AGENTS.md.
