neurocontainers

Imported from neurodesk/neurocontainers (AGENTS.md). Install upstream with npx skills add neurodesk/neurocontainers. Copyright stays with the author.

AGENTS.md - NeuroContainers Development Guide

Does and Don't

• Match the container version to the primary upstream software version: pad 7.1 to 7.1.0, preserve 7.1.3, and use the build date for rebuilds. Bind exact installed versions, source commits, and artifact digests to recipe variables. Declare auto_update.container_version for every sources policy; see version policies.
• always use {{ get_file("filename") }} to reference declared files in run directives instead of using wget or curl directly
• the home directory will not be available during container runtime! Files cannot be stored under /home if they are needed during runtime!
• make sure that every build.yaml recipe has a base64 encoded icon - first try to find the official icon, if none exist make one up based on the tool description.
• these are the available categories: functional imaging, fetal imaging, image reconstruction, spectroscopy, rodent imaging, data organisation, diffusion imaging, structural imaging, quantitative imaging, image segmentation, image registration, visualization, statistics, quality control, spine, electrophysiology, bids apps, machine learning, phase processing, molecular biology, hippocampus, body, shape analysis, cryo EM, programming, workflows.

Environment Setup

python3 -m venv env
source env/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -e .

Requires Python 3.10+. Uses uv as an alternative (uv run sf-build <name>).

Key Commands

Recipe Development

sf-init <name> <version>                  # Create new recipe from template
sf-generate <name>                        # Generate Dockerfile from recipe
sf-build <name>                           # Generate + build Docker image (does not run tests)
sf-login <name>                           # Build and drop into interactive shell
sf-test <name>                            # Run container tests (add --build to build first)
sf-make <recipe_dir>                      # Build SIF via BuildKit (no Docker daemon)
sf-cache <name>                           # Inspect/clear the recipe download cache

Build Options

sf-build <name> --architecture aarch64    # Build for ARM64
sf-build <name> --ignore-architectures    # Skip architecture checks
sf-build <name> --generate-release        # Generate release JSON after build

Direct builder usage

python -m builder generate <name> --recreate       # Generate Dockerfile only
python -m builder stage <name> --recreate          # Generate Dockerfile + stage files
python -m builder test <name> --recreate --build   # Build + run smoke test

Validation (Run Before Committing)

codespell .                                          # Spell check
source env/bin/activate && pytest builder/tests      # Builder + workflow unit tests
source env/bin/activate && ./workflows/test_all.sh   # Validate + generate every recipe
python3 builder/validation.py recipes/<name>/build.yaml  # Validate single recipe

Go tests live alongside their sources: cd dashboard && go test ./... and cd builder && go test docker-save-to-simg.go docker-save-to-simg_test.go.

Recipe Format (build.yaml)

Recipes are YAML files with Jinja2 templating. Key sections:

name: toolname
version: 1.0.0

copyright:
  - license: MIT                    # SPDX identifier required
    url: https://example.com/license

architectures:
  - x86_64
  - aarch64                         # Optional ARM64 support

variables:                          # Template variables with conditional logic
  download_url:
    try:
      - value: "https://example.com/tool-x86_64.tar.gz"
        condition: arch=="x86_64"
      - value: "https://example.com/tool-aarch64.tar.gz"
        condition: arch=="aarch64"

build:
  kind: neurodocker                 # Only supported builder kind
  base-image: ubuntu:24.04          # Debian/Ubuntu/RedHat base images
  pkg-manager: apt                  # apt or rpm

  directives:
    - environment:                  # Set env vars
        DEBIAN_FRONTEND: noninteractive
    - install:                      # System packages (via apt/rpm)
        - curl
        - wget
    - template:                     # NeuroDocker templates (ants, fsl, etc.)
        name: ants
        version: "{{ context.version }}"
    - workdir: /opt/tool            # Set working directory
    - run:                          # Shell commands (joined with &&)
        - curl -fLO https://example.com/tool.tar.gz
        - tar -xzf tool.tar.gz
        - rm tool.tar.gz
    - copy:                         # Copy declared files into container
        - script.sh
        - /opt/

deploy:
  bins:                             # Individual executables to expose
    - toolname
  path:                             # Directories with executables
    - /opt/tool/bin

files:                              # Declare files for build context
  - name: script.sh
    filename: script.sh             # Local file next to build.yaml
  - name: config.txt
    contents: |                     # Inline content
      key=value

categories:                         # NeuroDesk UI categories
  - "structural imaging"

readme: |                           # Inline documentation (supports Jinja2)
  ## toolname/{{ context.version }}
  Description here.

icon: "data:image/png;base64,..."   # Required - base64 data URI

Supported directives

builder/recipe.py accepts exactly these; anything else raises unsupported directive. The six shown above are the common ones:

Every directive also accepts a condition: key for architecture- or variant-specific behaviour.

fulltest.yaml is the only supported recipe test format. test.yaml, test.sh, and tests: blocks embedded in build.yaml are legacy and are read by nothing.

Release Test Format (fulltest.yaml)

Every new recipe should include recipes/<name>/fulltest.yaml for release/runtime smoke testing:

name: toolname
version: 1.0.0

tests:
  - name: tool launcher available
    description: Verify the main runtime entrypoint is installed.
    command: command -v toolname
    expected_output_contains: "toolname"

A fulltest always tests the container the recipe builds now, so name: must match the recipe directory and version: must match build.yaml. Bump both together; an older release is never substituted for a version that has not been built yet. Where a test needs the version — a --version string, an install path — write ${version} instead of spelling it out, so the assertion follows the recipe when the installed version follows the container version:

- name: reports its version
  command: toolname --version
  expected_output_contains: "${version}"

For independent source versions, declare a scalar such as upstream_version: in the fulltest, use ${upstream_version} in software assertions, and bind the source target's fulltest_variable to that scalar. The suite's version: still identifies the container release.

Do not add a container: key. The artifact under test is resolved from releases/<name>/<version>.json, so a hardcoded SIF name only ever goes stale; the runner rejects one that disagrees with the release metadata. The single exception is a test that genuinely needs a historical image, which pins it explicitly:

container: toolname_1.0.0_20250101.simg
pin_container: true

Testing Guidance

• Prefer tests that exercise shared builder behavior, workflow contracts, schema validation, or real container/runtime behavior.
• Every new recipe must include a focused recipes/<name>/fulltest.yaml unless there is a documented reason it cannot be tested. The test should prove the user-visible runtime contract, such as the main launcher, version/package metadata, required assets, desktop entry, or a small real command.
• For recipe-only fixes, usually validate the recipe and regenerate the Dockerfile; add or update fulltest.yaml only when it checks behavior a user or CI actually relies on.
• Do not add one-off regression tests that only match hardcoded strings already visible in git history or in the same recipe diff. These tests are brittle, duplicate version control, and create maintenance noise without proving behavior.
• Avoid recipe-specific Python tests unless the recipe exposes a reusable bug class or the assertion is meaningfully stronger than builder/validation.py plus Dockerfile generation.

Validation Schema

Recipe validation (builder/validation.py) enforces:

• Architectures: Must be x86_64 or aarch64
• Categories: Must match the 26-entry CATEGORIES list in builder/validation.py (reproduced under "Does and Don't" above)
• Licenses: Should use SPDX identifiers
• Required fields: name, version, architectures, build (with kind, base-image, pkg-manager, directives), categories, icon
• Optional: deploy — omitting it is valid, though most recipes want it

icon and categories are required and are the two most common validation failures on a new recipe: sf-init does not scaffold either, so add both before running validation.

The validation schema matches the Zod schema from neurocontainers-ui.

Code Conventions

YAML Recipes

• 2-space indentation
• Use Jinja2 {{ context.version }} and {{ context.variable_name }} for templating
• Use {{ get_file("filename") }} to reference declared files in run directives. This is the preferred method for downloading files (over wget or curl) because it utilizes the builder's local caching.
• Combine run directives where possible (each becomes a single Docker layer)
• Clean up temporary files in the same layer they are created
• Pin exact versions for reproducibility
• Use HTTPS for all downloads

Python Code

• PEP 8 style
• Type hints (Python 3.10+ style)
• Standard import order: stdlib, third-party, local

Commit Messages

• Descriptive, explain what and why
• Reference issue numbers when applicable (Fix #123: ...)
• Skip CI with [skipci] in commit message when needed

Common Development Tasks

Add a New Container Recipe

1. sf-init <toolname> <version> to scaffold
2. Edit recipes/<toolname>/build.yaml with build instructions, and replace the TODO placeholders. Add categories: and icon: — sf-init omits both and validation fails without them.
3. Configure auto_update with explicit targets for the installed software and shared dependencies. Bind source variables in install commands or declared downloads; derive the container label from the primary software version. Dependency-only updates retain that label and use the build date. For rolling sources, pin and track full commits or artifact digests. Locally maintained recipes use a sources policy with local repository inputs. Every recipe must pass python -m builder.audit_updates; manual and notification-only policies fail. Read automatic update policies when selecting a provider, tracking compiled artifacts or adding shared macros. OpenRecon consumers must include its four shared dependency sources.
4. Create recipes/<toolname>/fulltest.yaml with focused runtime smoke tests
5. Validate: python -m builder generate <toolname> --recreate
6. Build and test: sf-build <toolname> then sf-test <toolname>, or sf-login <toolname> for interactive debugging

Update a Container Version

1. Update version field in build.yaml
2. Update any version-specific URLs or checksums
3. Validate and rebuild

Debug a Failing Build

1. python -m builder generate <name> --recreate to inspect generated Dockerfile
2. Read the Dockerfile at build/<name>/<name>_<version>.Dockerfile
3. sf-login <name> to get an interactive shell in the built container
4. For CI failures, check GitHub Actions logs

Validate All Recipes

./workflows/test_all.sh              # Full validation + Dockerfile generation
./workflows/validate_all.sh          # Schema validation only (subset of test_all.sh)

Important Notes

• Container builds can take 1-60+ minutes depending on complexity. Do not cancel prematurely.
• Some existing recipes have known YAML issues or broken builds from upstream changes. Focus on your changes rather than fixing unrelated existing issues.
• The template recipe (recipes/template/) serves as the reference example.
• Builder includes local templates for the subset used by NeuroContainers recipes.
• The deploy section controls how tools appear as loadable modules in NeuroDesk via Transparent Singularity.
• Release files in releases/ are auto-generated; do not edit manually.