sync-guide

Imported from rishabhsai/notionbench (evals/build-workers-002-output-schema-robust/fixture/workspace/.agents/skills/sync-guide/SKILL.md). Install upstream with npx skills add rishabhsai/notionbench --skill sync-guide. Copyright stays with the author.

What is a Sync?

A sync is a recurring execute function that returns data changes to populate a Notion database. The runtime calls execute in a loop:

const db = worker.database("myDb", {
  type: "managed",
  initialTitle: "My Data",
  primaryKeyProperty: "ID",
  schema: {
    properties: {
      Name: Schema.title(),
      ID: Schema.richText(),
    },
  },
});

worker.sync("mySync", {
  database: db,
  execute: async (state, { notion }) => ({
    changes: [
      { type: "upsert", key: "1", properties: { Name: Builder.title("Item 1"), ID: Builder.richText("1") } },
    ],
    hasMore: false,
    nextState: undefined,
  }),
});

Each call returns { changes, hasMore, nextState }. If hasMore is true, the runtime calls execute again with nextState. This continues until hasMore is false, completing a cycle. The next cycle begins at the scheduled interval with the state from the end of the previous cycle.

Imports:

import { Worker } from "@notionhq/workers";
import * as Builder from "@notionhq/workers/builder";
import * as Schema from "@notionhq/workers/schema";

Decision Framework

Step 1: Choose an Architecture

The deciding factor is API capability and dataset size. Two tiers:

Simple replace sync: One sync returns the full dataset each cycle. After the final hasMore: false, any records not seen are deleted automatically. Use when the dataset is small enough to re-fetch entirely.

Backfill + delta pair: Two syncs share a single database. The backfill sync (mode: "replace", schedule: "manual") re-fetches everything when triggered. The delta sync (mode: "incremental", frequent schedule) fetches only changes since the last run. This separates concerns cleanly — no bi-modal state machine, no backfill-to-delta transition bugs.

Step 2: Understand Your API's Pagination

Most APIs require paginating through results. Return batches of ~100 changes. Returning too many changes in one execute call will fail.

Backfill pagination (full dataset load):

1. Opaque cursor token — GraphQL endCursor, Stripe starting_after
2. Page number / offset — ?page=N&limit=100
3. Keyset (timestamp + id) — WHERE created_at > X OR (created_at = X AND id > Y) — the gold standard for timestamp-sorted mutable data

Delta pagination (change-only loads, incremental mode):

1. Timestamp cursor — ?updated_since=<cursor> with consistency buffer
2. Keyset on updated_at + id — same keyset pattern on the modification timestamp
3. Event/changelog feed — GET /events?after=<eventId>
4. Same opaque cursor — when the API sorts by updated_at, the backfill cursor works for delta too

Step 3: Consistency Buffer (Delta Syncs)

APIs tend to be eventually consistent. A record that was just written or updated may not appear in query results immediately. Since the cursor never resets in incremental mode, if it advances past a record that hasn't been indexed yet, that record is skipped permanently. Lag the cursor 10-60 seconds behind "now":

const bufferMs = 15_000;
const maxCursor = new Date(Date.now() - bufferMs).toISOString();
const nextCursor = records.length > 0
  ? min(lastRecord.updatedAt, maxCursor)
  : maxCursor;

Step 4: Deletion Strategies

1. Backfill sync (replace mode): free — unseen records are auto-deleted each cycle. This is the primary mechanism for handling deletes when the API has no delete signal.
2. Delta sync with delete API: emit { type: "delete", key } markers. If the delete signal comes from a separate endpoint (audit log, archived filter), use the flip-flop pattern: run the main delta stream until caught up (hasMore: false), then switch to the delete stream for a cycle, then back. Both cursors persist in state independently.
3. No delete API, large dataset: rely on the backfill sync's replace-mode mark-and-sweep. Trigger the backfill manually or on a slow schedule to clean up stale records.

Replace Mode

Simple: fetch everything, return it all, let the runtime handle deletes. Use as a standalone sync for small sources, or as the backfill half of a backfill+delta pair.

const db = worker.database("records", {
  type: "managed",
  initialTitle: "Records",
  primaryKeyProperty: "ID",
  schema: {
    properties: { Name: Schema.title(), ID: Schema.richText() },
  },
});

const apiPacer = worker.pacer("myApi", {
  allowedRequests: 10,
  intervalMs: 1000,
});

worker.sync("recordsBackfill", {
  database: db,
  mode: "replace",
  schedule: "manual",  // trigger manually or on a slow schedule
  execute: async (state) => {
    const page = state?.page ?? 1;
    await apiPacer.wait();
    const { items, totalPages } = await fetchPage(page, 100);
    const hasMore = page < totalPages;
    return {
      changes: items.map((item) => ({
        type: "upsert" as const,
        key: item.id,
        properties: { Name: Builder.title(item.name), ID: Builder.richText(item.id) },
      })),
      hasMore,
      nextState: hasMore ? { page: page + 1 } : undefined,
    };
  },
});

See examples/replace-simple.ts and examples/replace-paginated.ts for complete working examples.

Incremental Mode (Delta Sync)

The delta sync fetches only changes since the last run. When paired with a replace-mode backfill sync on the same database, this replaces the old bi-modal single-sync pattern.

// Reuses the same `db` and `apiPacer` from above

worker.sync("recordsDelta", {
  database: db,
  mode: "incremental",
  schedule: "5m",
  execute: async (state: { cursor: string } | undefined) => {
    const cursor = state?.cursor ?? new Date(0).toISOString();
    const bufferTs = new Date(Date.now() - 15_000).toISOString();

    await apiPacer.wait();
    const { items, nextCursor } = await fetchChanges(cursor);
    const done = !nextCursor;

    return {
      changes: items.map(toUpsert),
      hasMore: !done,
      nextState: {
        cursor: done ? min(nextCursor ?? cursor, bufferTs) : nextCursor,
      },
    };
  },
});

Key points:

• The delta sync's state is simple — just a cursor. No phase discrimination needed.
• The backfill sync (replace mode) handles the initial full load and periodic cleanup of deleted records.
• Both syncs write to the same database via the shared db handle.
• The pacer is shared between syncs — the server apportions the budget evenly.

See examples/incremental-basic.ts, examples/incremental-bimodal.ts, and examples/incremental-events.ts for complete patterns.

Schema Reference

Define the Notion database shape with Schema types and build values with Builder:

Relations use the related database key. Two-way relations are configured the same way:

Schema.relation("otherDatabase", { twoWay: true, relatedPropertyName: "Back Link" })

Row-level icons and page content:

changes: [{
  type: "upsert", key: "1",
  properties: { ... },
  icon: Builder.emojiIcon("🎯"),               // or Builder.notionIcon("rocket", "blue")
  pageContentMarkdown: "## Details\nSome text", // Markdown body for the page
}]

Common Mistakes

1. Not using a pacer — every API call inside execute should be preceded by await apiPacer.wait(). Without it, syncs will hit rate limits and fail.
2. Missing consistency buffer on delta syncs — the cursor will permanently skip records not yet indexed in eventually consistent APIs.
3. Not paginating — returning too many changes at once. Start with batches of ~100.
4. Using replace mode for large datasets — if the API supports change tracking, pair a replace-mode backfill sync with an incremental delta sync instead of re-fetching everything each cycle.
5. Cursor that doesn't advance — infinite loop. Ensure nextState changes between iterations.
6. Forgetting first-run handling — state is undefined on first call. Use state?.cursor ?? null.
7. Forgetting that backfill + delta share a database — both syncs must use the same worker.database() handle and the same key/properties shape.
8. Not triggering the backfill sync — the backfill sync with schedule: "manual" won't run automatically. Trigger it on deploy or periodically to clean up deleted records.
9. Empty select values — Schema.select() requires at least one option with a non-empty name. Schema.select([]) and { name: "" } are not supported.

CLI Commands for Sync Development

# Deploy
ntn workers deploy

# Preview (test without writing)
ntn workers sync trigger <key> --preview
ntn workers sync trigger <key> --preview --context '<json>'  # continue pagination

# Trigger a sync run
ntn workers sync trigger <key>

# Check sync status
ntn workers sync status

# View run logs
ntn workers runs list
ntn workers runs list --plain | head -n1 | cut -f1 | xargs -I{} ntn workers runs logs {}

# Reset state (full re-backfill)
ntn workers sync state reset <key>

# Manage secrets
ntn workers env set KEY=value
ntn workers env push

API Patterns Reference

See api-pagination-patterns.md for detailed strategies drawn from production syncs with Salesforce, Stripe, HubSpot, GitHub, and ServiceNow.