Imported from lukasspires/github-copilot-fullstack-kit (
.agents/skills/etl-pipeline/SKILL.md). Install upstream withnpx skills add lukasspires/github-copilot-fullstack-kit --skill etl-pipeline. Copyright stays with the author.
Reliable ETL pipeline
Use for changes to multi-stage contracts, orchestration, recovery or replay. Connector-only fixes use the matching API, file or scraping skill.
- Inspect affected runtime, scheduler, source/destination schema, keys, transformations and recovery paths; reuse established patterns and evidence.
- Consult affected items in the pipeline checklist. A small correction does not mandate new checkpoints, metrics, artifacts or a complete pipeline redesign.
- Preserve precision, audit, null/default and timezone semantics unless explicitly changed. For normalization of keys, investigate collisions, upsert behavior and existing historical rows before deciding whether migration/backfill is needed.
- Involve the architect for material architecture, shared contracts, migrations/backfills. Live replay/backfill requires corresponding authorization; local implementation/testing can continue independently.
- For new or affected recovery logic, establish idempotency, durable checkpoint ordering, partial-failure restart and bounds on memory/concurrency/retries. Add observability or run/recovery documentation where changed behavior needs it.
- Test the altered transformations and failure boundaries with representative offline fixtures. Use the reviewer for data-integrity or critical operational changes and report unresolved historical-data risks.