About

Imported from shipshitdev/skills (skills/incremental-fetch/SKILL.md). Install upstream with npx skills add shipshitdev/skills --skill incremental-fetch. Copyright stays with the author.

Incremental Fetch

Name: incremental-fetch
Author: shipshitdev

Build data pipelines that never lose progress and never re-fetch existing data.

The Two Watermarks Pattern

Track TWO cursors to support both forward and backward fetching:

Watermark	Purpose	API Parameter
`newest_id`	Fetch new data since last run	`since_id`
`oldest_id`	Backfill older data	`until_id`

A single watermark only fetches forward. Two watermarks enable:

Regular runs: fetch NEW data (since newest_id)
Backfill runs: fetch OLD data (until oldest_id)
No overlap, no gaps

Critical: Data vs Watermark Saving

These are different operations with different timing:

What	When to Save	Why
Data records	After EACH page	Resilience: interrupted on page 47? Keep 46 pages
Watermarks	ONCE at end of run	Correctness: only commit progress after full success

fetch page 1 → save records → fetch page 2 → save records → ... → update watermarks

Workflow Decision Tree

First run (no watermarks)?
├── YES → Full fetch (no since_id, no until_id)
└── NO → Backfill flag set?
    ├── YES → Backfill mode (until_id = oldest_id)
    └── NO → Update mode (since_id = newest_id)

Implementation Checklist

Database: Create ingestion_state table (see patterns.md)
Fetch loop: Insert records immediately after each API page
Watermark tracking: Track newest/oldest IDs seen in this run
Watermark update: Save watermarks ONCE at end of successful run
Retry: Exponential backoff with jitter
Rate limits: Wait for reset or skip and record for next run

Pagination Types

This pattern works best with ID-based pagination (numeric IDs that can be compared). For other pagination types:

Type	Adaptation
Cursor/token	Store cursor string instead of ID; can't compare numerically
Timestamp	Use `last_timestamp` column; compare as dates
Offset/limit	Store page number; resume from last saved page

See references/patterns.md for schemas and code examples.

Gotchas

Save watermarks only after full success. If the process crashes mid-run, unsaved watermarks mean the next run re-fetches and deduplicates from scratch — no data loss, but potentially slow. Saving watermarks mid-run causes permanent gaps.
Newest ID may not equal the highest numeric ID. Some APIs return IDs that are not monotonically increasing (e.g., snowflake IDs with clock drift). Always compare using the API's own ordering guarantees, not numeric comparison.
Backfill mode must not overwrite the newest_id. A backfill run extends history backward; it should update only oldest_id. Overwriting newest_id during backfill causes duplicate fetches on the next forward update run.
Rate-limit headers vary by API. Twitter uses x-rate-limit-reset; others use Retry-After. Check the specific API's response headers before implementing wait logic.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/shipshitdev-skills-incremental-fetch/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

shipshitdev-skills-incremental-fetch.ocm.jsonjson

{
  "ocm": "1",
  "id": "shipshitdev-skills-incremental-fetch",
  "kind": "skill",
  "name": "incremental-fetch",
  "description": "Guides construction of resilient data ingestion pipelines from paginated APIs. Activates on: \"ingest data from API\", \"pull tweets\", \"fetch historical data\", \"sync from X\", \"build a data pipeline\", \"fetch without re-downloading\", \"resume the download\", \"backfill older data\". NOT for: simple one-shot API calls, websocket/streaming connections, file downloads, or APIs without pagination.",
  "publisher": "shipshitdev",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "coding",
      "hr"
    ],
    "tags": [
      "skill-md",
      "data-ingestion",
      "api",
      "pagination",
      "skills-sh"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Guides construction of resilient data ingestion pipelines from paginated APIs. Activates on: \"ingest data from API\", \"pull tweets\", \"fetch historical data\", \"sync from X\", \"build a data pipeline\", \"fetch without re-downloading\", \"resume the download\", \"backfill older data\". NOT for: simple one-shot API calls, websocket/streaming connections, file downloads, or APIs without pagination."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/shipshitdev/skills",
      "path": "skills/incremental-fetch/SKILL.md",
      "ref": "HEAD",
      "url": "https://github.com/shipshitdev/skills/blob/HEAD/skills/incremental-fetch/SKILL.md",
      "key": "shipshitdev/skills/skills/incremental-fetch/SKILL.md"
    }
  },
  "instructions": "# Incremental Fetch\n\nBuild data pipelines that never lose progress and never re-fetch existing data.\n\n## The Two Watermarks Pattern\n\nTrack TWO cursors to support both forward and backward fetching:\n\n| Watermark | Purpose | API Parameter |\n|-----------|---------|---------------|\n| `newest_id` | Fetch new data since last run | `since_id` |\n| `oldest_id` | Backfill older data | `until_id` |\n\nA single watermark only fetches forward. Two watermarks enable:\n\n- Regular runs: fetch NEW data (since `newest_id`)\n- Backfill runs: fetch OLD data (until `oldest_id`)\n- No overlap, no gaps\n\n## Critical: Data",
  "cost": {
    "context_tokens": 761
  }
}

Fetch it by URL: GET /api/v1/registry/shipshitdev-skills-incremental-fetch/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.