substreams-dev

Imported from XuechunHHH/substreams-preflight (skills/substreams-dev/SKILL.md). Install upstream with npx skills add XuechunHHH/substreams-preflight --skill substreams-dev. Copyright stays with the author (Apache-2.0).

Substreams Development Expert

Expert assistant for cross-cutting Substreams development: manifests, module graphs, protobuf, build/run, and performance. Chain-specific contract/program work lives in dedicated skills.

Skill routing (load the right skill)

Always establish the target chain early. If the user has not said Ethereum vs Solana (or another family), ask before writing code — then follow the chain skill for implementation details.

Core Concepts

Substreams indexes blockchain data through composable Rust modules compiled to WASM, executed in parallel by the engine, with cursor-based reorg handling.

1. Manifest (substreams.yaml): modules, networks, dependencies
2. Modules: map (transform), store (aggregate), index (filter)
3. Protobuf: typed schemas for inputs and outputs
4. WASM: Rust compiled to wasm32-unknown-unknown

Project Structure

Generic layout — both EVM and Solana use build.rs for domain protobufs via prost_build. EVM additionally uses abi/ + Abigen in the same build.rs; Solana does not.

my-substreams/
├── substreams.yaml              # Manifest (manual) — must include protobuf: + binaries:
├── README.md                    # Package docs for substreams.dev registry (manual)
├── Cargo.toml                   # Rust dependencies (manual)
├── build.rs                     # prost_build (required); EVM also runs Abigen here
├── proto/events.proto           # Schema definitions (manual)
├── src/
│   ├── lib.rs                   # Rust module code + mod pb { include!(OUT_DIR) } (manual)
│   └── pb/                      # Optional: only if using substreams protogen (auto)
└── target/                      # Build output (gitignored)

Do not hand-write generated proto Rust. Prefer build.rs + include!(concat!(env!("OUT_DIR"), "/…")) (Solana T5.x / most EVM examples). If you use substreams protogen, the src/pb/ tree is auto-generated — never create it by hand.

substreams protogen  # Proto bindings only (fast iteration)
substreams build     # Full WASM + spkg

Chain-specific scaffolding (full Solana / EVM Cargo + build.rs + packable manifest): substreams-solana / substreams-ethereum.

Prerequisites

• substreams: core CLI for building, running, deploying
• buf: required by substreams build for protobuf codegen
• Rust: rustup target add wasm32-unknown-unknown

Authentication

substreams run against hosted endpoints requires auth. Get an API key from The Graph Market.

substreams auth   # interactive; exchanges the key and stores the token locally

Alternative: export SUBSTREAMS_API_KEY="…" (or SUBSTREAMS_API_TOKEN="<jwt>" for a bearer token directly).

If you send the user to a browser login, treat it like Portal device login: show the link, end the turn, and wait for the user to confirm — do not background-poll or busy-wait.

This is data-plane auth, not Portal auth. SUBSTREAMS_API_KEY / SUBSTREAMS_API_TOKEN authenticate substreams run against the streaming data endpoints. The StreamingFast Portal admin API (billing, usage, hosted deployments) is separate — see thegraph-market-api. A Portal bearer token will not authenticate substreams run. Reach for that skill only for plan, quota, usage, or deployment status.

Pre-flight: Clarifying Under-Specified Requests

Before writing code, confirm the items below. If any is missing or ambiguous, ask — do not assume silently.

How to ask (MANDATORY for all Substreams skills)

1. One question per turn — never batch a questionnaire. Ask, wait, then ask the next.
2. Offer explicit choices — selectable options (platform choice UI when available; else a numbered list).
3. Always include a custom-answer choice — the last option must allow free text ("Other / enter a custom answer").
4. Skip what is already known — do not re-ask what the user stated.
5. Order matters — walk the list top-to-bottom.

Bad: one message with 6 free-text questions and no options. Good: "Which chain?" → options → answer → next turn "Where should output go?" → options including custom.

Output destination — ask as its own turn (ALWAYS offer all of these)

SQL is two destinations, not one: if the user says only "SQL", ask in a separate turn self-managed vs StreamingFast hosted (plus custom). Hosted still needs a correct SQL module (substreams-sql).

Once the chain is known, load and follow substreams-ethereum or substreams-solana. Keep this skill for module graph design, manifest rules, and performance. Chain and sink skills use the same one-question + choices + custom protocol.

If the prompt is concrete and complete, skip the checklist and build immediately.

Mechanical preflight

Before generating a project, record the confirmed requirements in substreams.intent.json:

{
  "chain": "ethereum-mainnet",
  "target": "ERC-721 contracts",
  "data": "mint events",
  "destination": "The Graph / graph_out",
  "range": "18000000-18000100"
}

Do not create this file by guessing missing values. Ask the required single clarification first. This guard supports Ethereum/EVM and Solana only; reject or request an authoritative integration reference for another chain.

Before substreams build, run:

npm run preflight -- --intent substreams.intent.json --project .

For graph_out, this verifies the critical EntityChanges.entity_changes = 5 and Field.new_value = 3 wire tags that normal build/graph validation does not catch. If it fails, fix the declared contract before building; do not deploy.

Discovering Existing Packages (Registry Search API)

Before building from scratch, search the registry — an existing package may already index the data you need. Public, no auth required.

Endpoint: GET https://substreams.dev/v1/registry/packages

curl "https://substreams.dev/v1/registry/packages?query=uniswap&page_size=5"

Response shape — fields are omitted, not empty. This is proto3 JSON: default-valued and absent fields do not appear at all.

{
  "packages": [
    {
      "name": "uniswap_v3",
      "slug": "uniswap-v3",
      "organization": { "name": "Uniswap", "slug": "uniswap" },
      "repository": "https://github.com/...",
      "downloads": 7,
      "releaseCount": 1,
      "latestVersion": "v0.1.0",
      "network": "mainnet",
      "reference": "uniswap-v3@v0.1.0",
      "spkg": "https://spkg.io/v1/packages/uniswap-v3/v0.1.0"
    }
  ],
  "hasMore": true
}

Parse defensively — every one of these is real:

• hasMore is absent when there are no more pages (not false). Treat missing as "done".
• Zero results returns {} — not {"packages": []}. Treat missing packages as an empty list.
• network, organization, and repository are omitted when absent. ethereum-common has no network field.
• name and slug differ: name may use underscores (uniswap_v3), slug uses hyphens (uniswap-v3). URLs are built from the slug.

Do not add network=mainnet reflexively. It excludes packages that have no network field — including ethereum-common, the foundational package this skill tells you to import. ?query=ethereum-common returns it; ?query=ethereum-common&network=mainnet returns {}.

spkg vs reference — pick the right one:

• spkg — full package URL (https://spkg.io/v1/packages/<slug>/<version>). Use this for execution and for imports:: substreams run <spkg> <module>, substreams gui <spkg>, substreams info <spkg>, imports: { eth_common: <spkg> }.
• reference — short <slug>@<version> notation. Display only. The CLI rewrites name@version to https://substreams.dev/v1/packages/<name>/<version>, which returns an HTML 404. Working download hosts are spkg.io and api.substreams.dev only. There is no @latest tag — pin the version from latestVersion / spkg.

Both are absent when no release version is known.

Rate limits: the API is rate-limited per IP. Over-limit requests return 429 with a Retry-After header — honor it and back off. The full contract is published at GET https://substreams.dev/v1/registry/openapi.yaml (source of truth for fields; it does not document the rate limits).

Creating a New Project

Migrating an existing project? Load substreams-convert if porting a subgraph or Solana/EVM contract rather than starting fresh.

1. Initialize: substreams init, or write the manifest manually
2. Define schema: .proto files for your data structures
3. Implement modules: Rust handlers in src/lib.rs
4. Build: substreams build → .spkg
5. Test: substreams run over a small range (~1000 blocks)
6. Document: README.md for the registry (below)
7. Publish: to the registry, or deploy as a service

README for substreams.dev Registry

Every published package must ship a README.md — it is the package's front page on substreams.dev, and its presence silences the README (package.doc) not found warning.

Required sections: title (matching package.name), one-line description, Overview (2-3 sentences: what data, what chain, intended use), Modules table (every name: from the manifest, with kind + output type + description — consumers need this to know what to substreams run), Prerequisites, and a Quick Start with a real substreams run command over a real block range (not a placeholder).

Do NOT add "Contributing" or "License" sections — the registry pulls license from the manifest.

Publishing

substreams registry verify ./substreams.yaml   # optional preflight
substreams build
substreams publish ./substreams.yaml --yes     # or path to .spkg

Hosted SQL sink package source (important): after publish, for StreamingFast hosted Deploy prefer:

https://api.substreams.dev/v1/packages/<package-name>/<version>

Verify the URL returns a binary .spkg (size ≈ local file), not HTML. Prefer this as spkg.url over substreams_dev_id alone — some hosted runners mis-parse substreams-dev://name@version. Full flow: substreams-hosted-sink + thegraph-market-api.

Module Types

Full details: references/module-types.md.

Map — transforms input to output:

- name: map_events
  kind: map
  inputs:
    - source: sf.ethereum.type.v2.Block
  output:
    type: proto:my.types.Events

Store — aggregates across blocks:

- name: store_totals
  kind: store
  updatePolicy: add
  valueType: int64
  inputs:
    - map: map_events

Store inputs use block form, never inline commas. - store: x, mode: deltas is a YAML parse error — a plain scalar may not contain ": ". Write:

inputs:
  - store: store_totals
    mode: deltas      # or: get

Index (kind: blockIndex) — emits per-block Keys so downstream modules can skip blocks. The handler is #[substreams::handlers::map] returning Keys. See "Block & Transaction Filtering" below.

initialBlock

modules:
  - name: map_events
    kind: map
    initialBlock: 18000000   # ✅ start of your test/data range
    # NOT: 12369621          # ❌ protocol genesis — forces full backfill on every run

Pin initialBlock to the first block your downstream consumer actually needs. The runtime starts at max(--start-block, initialBlock) and walks forward; stores must catch up from initialBlock on each cold start, so a deep genesis pin turns a 100-block test into a multi-hour backfill.

Block & Transaction Filtering (Cost-Critical)

Whenever a Substreams targets only a subset of blocks — a specific contract, program, event signature, account, or transaction type — add a block index filter. Be aggressive about this. It is the biggest win available for both your iteration speed and your bill: you are billed for blocks the engine actually processes, so a contract active in 0.5% of blocks costs roughly 0.5% as much, and backfills that would take hours finish in minutes.

The pattern (three pieces):

# 1. Index module — kind `blockIndex`, output `Keys`
- name: index_events
  kind: blockIndex
  inputs:
    - map: all_events
  output:
    type: proto:sf.substreams.index.v1.Keys

# 2. Consuming module — declares blockFilter to skip non-matching blocks
- name: filtered_events
  kind: map
  blockFilter:
    module: index_events
    query:
      params: true          # SQE query from `params` (or: string: "<expr>")
  inputs:
    - params: string
    - map: all_events
  output:
    type: proto:my.types.Events

The index handler is a map handler returning Keys (a repeated string of labels you choose):

#[substreams::handlers::map]
fn index_events(events: Events) -> Result<Keys, Error> {
    let mut keys = Keys::default();
    for e in events.events {
        if let Some(log) = e.log {
            if let Some(t0) = log.topics.get(0) {
                keys.keys.push(format!("evt_sig:0x{}", Hex::encode(t0)));
            }
            keys.keys.push(format!("evt_addr:0x{}", Hex::encode(&log.address)));
        }
    }
    Ok(keys)
}

Query (SQE): boolean expression over keys — &&, ||, - (not), ( ) grouping. A bare term must match a key exactly: "evt_addr:0xA && -evt_addr:0xspam".

Critical rules:

• The index does nothing automatically. Only a module with an explicit blockFilter gets blocks skipped — listing the index as a dependency is not enough.
• Query namespace must match the emitted key prefix exactly (evt_addr: vs address:), and values match by literal equality — use 0x-prefixed lowercase hex (EVM checksum/mixed-case addresses never match).
• Don't reinvent. Most chains ship a foundational package whose filtered_* modules already apply the blockFilter and return only matching records — depend on those via spkg URL (short name@version imports 404):

  imports:
    eth_common: https://spkg.io/v1/packages/ethereum-common/v0.3.3
  # then: map: eth_common:filtered_events

  You must override the default params query, or you silently emit the default's data. In-handler filtering is only needed when you roll your own blockFilter, or for Solana instruction-level filtering (transactions are pre-filtered, instructions within them are not).

Full guide (SQE syntax, params vs string, use inheritance, foundational indexes, decision flowchart): references/block-filtering.md.

Manifest Reference

Full spec: references/manifest-spec.md.

specVersion: v0.1.0
package:
  name: my-substreams
  version: 1.6.0
  url: https://github.com/myorg/my-substreams
  description: What this substreams does

protobuf:
  files:
    - events.proto
  importPaths:
    - ./proto

binaries:
  default:
    type: wasm/rust-v1
    file: ./target/wasm32-unknown-unknown/release/my_substreams.wasm

network: mainnet   # see references/networks.md

version requires a v prefix. Use v0.1.0, not 0.1.0. The error (version "0.1.0" should match Semver) is misleading — both are valid semver, but Substreams mandates the v form. Applies to package.version only.

Always set url + description, add a README.md, and never use package.doc. Omitting these produces a wall of non-fatal warnings on every build — confusing, looks broken, isn't. package.doc is deprecated; delete it if a scaffold includes it and write a README.md instead (the packager picks it up automatically). Module-level doc: fields are fine.

WASM bindgen shims — some crates (solana_program, alloy, chrono) emit bindgen imports under wasm32-unknown-unknown. To compile:

binaries:
  default:
    type: wasm/rust-v1+wasm-bindgen-shims

The shims allow compilation but do not implement the underlying functionality — avoid calling those imports at runtime.

Rust Module Development

Cargo.toml (cross-cutting)

Always set crate-type = ["cdylib"], release LTO, and matching prost / prost-types majors.

Pin prost / prost-types to 0.13 — do not bump to 0.14. substreams 0.7 requires prost ^0.13.3; bumping puts two prost copies in the tree and produces exactly the link errors below. This pin is intentionally behind crates.io latest.

Shared pitfalls:

• hex-literal uses a hyphen in Cargo.toml
• Missing prost-types = "0.13" when generated src/pb references Timestamp / Any
• Mix-major is not one failure mode. Solana mismatched pairs (substreams 0.6 + substreams-solana 0.15, or 0.7 + 0.14) still compile but pull two substreams copies — check cargo tree, realign to a matched pair (substreams-solana). Link / multiply-defined errors are more often dual prost (0.13 vs 0.14) or substreams-database-change 4 on a 0.6 tree — realign then rm -rf target && substreams build
• WASM bindgen errors → wasm/rust-v1+wasm-bindgen-shims and/or default-features = false on chrono etc.

Map / store shape (generic)

#[substreams::handlers::map]
pub fn map_events(/* chain block type */) -> Result<Events, Error> {
    // Transform one block → typed proto output
    Ok(Events::default())
}

#[substreams::handlers::store]
pub fn store_totals(events: Events, store: StoreAddInt64) {
    for item in events.items {
        store.add(0, &item.key, item.amount as i64);
    }
}

• EVM block type + log decoding + eth_call + token stores: substreams-ethereum
• Solana block type + walk_instructions + Anchor/SPL: substreams-solana
• Immutable per-key caches use set_if_not_exists stores, never a HashMap inside a map handler (RPC budget).

Best Practices

• Handle errors gracefully: return Result<T, Error>; never panic in handlers
• Log sparingly: excessive logging costs performance (substreams::log::info! / log::debug! — there is no warn!)
• Validate inputs: check for null/empty data before processing
• Test locally first: always substreams run before deploying
• Borrow, don't clone (below)

Performance

• Add a blockFilter to skip irrelevant blocks — by far the biggest cost lever (see above)
• Minimize store size — store only what you need
• --production-mode enables parallel execution for large ranges
• Module granularity — smaller, focused modules perform better; avoid deep dependency nesting

Avoid cloning large structures

The chain Block and its transactions are large. Cloning them in a handler is the most common self-inflicted performance problem.

• ❌ Never clone the Block — copies every transaction with it.
• ❌ Avoid cloning transactions — extract the few fields you need instead.
• ✅ Iterate by reference and borrow. The chain crates' accessors already yield references — take them as &T and copy only the small owned values (a hash string, an amount) into your proto output.
• ✅ Clone freely when it is genuinely small — a String key reused across an inner loop is fine, and cloning a Copy primitive like block.number is a no-op you need not think about.

Chain-specific iteration (EVM transactions() / logs_with_calls(), Solana walk_instructions()) belongs to substreams-ethereum and substreams-solana — follow those skills for the correct loop shape, which also determines which records you see, not just how fast you see them.

Measuring: substreams run executes your WASM server-side, so wall-clock time is dominated by network transfer and endpoint queueing, and a code change alters the module hash (cold cache) while the baseline ran warm. It is not a controlled A/B for handler CPU. To measure handler cost, benchmark at the WASM level — see substreams-testing.

Common Patterns

references/patterns.md — store aggregation, multi-module composition, parameterized modules, error handling.

• EVM event extraction / ABI / eth_call → substreams-ethereum
• Solana instructions / SPL / Anchor → substreams-solana
• Database sink patterns → substreams-sql

graph_out Modules (The Graph / Subgraph Output)

Also load the substreams-sink skill when building a graph_out module — it has the full working example.

These are NOT interchangeable. Using DatabaseChanges in a graph_out module (or vice versa) compiles but produces a pipeline that fails or emits garbage.

Do not rely on the substreams-entity-change crate for modern substreams 0.7 pipelines. v1 pins prost ^0.11 (conflicts with prost 0.13); v2.0.0 has prost ^0.13 but still depends on substreams ^0.6, so it pulls a second substreams copy into a 0.7 tree. Inline the proto instead. Re-check crates.io before changing this advice.

proto/entity.proto (exact package name required):

syntax = "proto3";

package sf.substreams.sink.entity.v1;

option go_package = "github.com/streamingfast/substreams-sink-entity-changes/pb/sf/substreams/sink/entity/v1;pbentity";

message EntityChanges {
  repeated EntityChange entity_changes = 5;   // NOT 1 — see the warning below
}

message EntityChange {
  string entity = 1;
  string id = 2;
  // Deprecated, this is not used within `graph-node`.
  uint64 ordinal = 3;
  enum Operation {
    OPERATION_UNSPECIFIED = 0;
    OPERATION_CREATE = 1;
    OPERATION_UPDATE = 2;
    OPERATION_DELETE = 3;
    OPERATION_FINAL = 4;
  }
  Operation operation = 4;
  repeated Field fields = 5;
}

message Value {
  oneof typed {
    int32 int32 = 1;
    string bigdecimal = 2;
    string bigint = 3;
    string string = 4;
    string bytes = 5;
    bool bool = 6;
    int64 timestamp = 7;
    Array array = 10;
  }
}

message Array {
  repeated Value value = 1;
}

message Field {
  string name = 1;
  optional Value new_value = 3;
  // Deprecated, this is not used within `graph-node`.
  optional Value old_value = 5;
}

Wire compatibility — field numbers are not cosmetic. Copy this proto verbatim from the canonical source. Package name, message names, field numbers, and field types must all match exactly.

The failure mode is silent: entity_changes is tag 5, not 1. Number it 1 and your module still compiles, still runs, and still emits data — but Graph Node reads tag 5, finds nothing, and ingests zero entities with no error. Likewise Operation must use the OPERATION_* names (a bare UNSET/CREATE enum changes the generated Rust variants), and bytes is a string field, not bytes.

The proven-working variant in examples/T3.2-cross-dex-volume/proto/entity.proto omits the deprecated old_value and unused timestamp — safe, since dropping unused optional fields does not shift the tags that matter.

output:
  type: proto:sf.substreams.sink.entity.v1.EntityChanges

use crate::pb::sf::substreams::sink::entity::v1::{EntityChange, EntityChanges, Field};
use crate::pb::sf::substreams::sink::entity::v1::entity_change::Operation;

Querying Chain Head Block

Requires data-plane auth (substreams auth / SUBSTREAMS_API_TOKEN).

# map_clocks emits sf.substreams.v1.Clock for every block; -s -1 starts at chain head.
# Do NOT combine -s -1 with a relative -t +N (the CLI requires an absolute start for +stop).
# Ctrl-C after the first line, or pipe through head.
substreams run https://spkg.io/v1/packages/common/v0.1.0 map_clocks \
  -s -1 --network mainnet -o jsonl

Read the first line for the head clock (number / hash). Default endpoint for a network: substreams tools default-endpoint <network>.

Using firecore: firecore tools firehose-client <network-id-alias-or-host> -o json -- -1

Debugging

1. Validate the graph: substreams graph to visualize dependencies
2. Test small: run 100-1000 blocks, inspect output carefully
3. Check logs: WASM panics, protobuf decode errors
4. Verify schema: proto types match expected data
5. Review inputs: confirm input modules produce correct data
6. Check initialBlock: see the initialBlock guidance above — a genesis pin makes local runs impractical

Troubleshooting

Non-fatal package warnings (package.doc deprecated, README (package.doc) not found, URL (package.url) is not set, Description (package.description) is not set): all four clear with a package: block that sets url + description, a sibling README.md, and no doc: field. Builds still succeed. See "Manifest Reference".

Build fails:

• version "x.y.z" should match Semver → add the v prefix to package.version
• rustup target add wasm32-unknown-unknown
• Ensure buf is installed (required for proto generation)
• Add protobuf.excludePaths with sf/substreams and google when importing spkgs
• Ensure the binary path in the manifest matches the build output

Linking errors ("symbol multiply defined", "failed to load bitcode") — almost always crate version mismatch:

• rm -rf target && substreams build
• Check for two substreams or two prost copies (cargo tree -d)
• Common link breakers: dual prost (pin 0.13), or substreams 0.6 + substreams-database-change 4 (needs 0.7)
• Dual substreams from a mismatched Solana pair often still links — fix the pins anyway (substreams-solana)

Missing method on ABI-generated types ("no method named decode"): add use substreams_ethereum::Event; and ensure ethabi = "18".

spkg import 404s:

• Prefer full URLs: https://spkg.io/v1/packages/<slug>/<version> or https://api.substreams.dev/v1/packages/<slug>/<version>. Short name@version rewrites to https://substreams.dev/v1/packages/... and 404s (HTML site, not the package API).
• Use the substreams-ethereum spkg, NOT sf-ethereum (doesn't exist)
• Verify the release exists (registry latestVersion)
• Check the slug — registry slugs use hyphens (ethereum-common), while name may use underscores

Empty output:

• Confirm initialBlock is at or before the first relevant block
• Check the module isn't filtered out by an upstream index (blockFilter query mismatch — lowercase hex?)
• Verify input data exists in the block range

Resources

• Official Documentation
• Module Types · Manifest Spec · Block Filtering · Patterns · Networks
• substreams-ethereum — EVM contracts · substreams-solana — Solana programs
• Discord · GitHub Issues