Imported from ColinConwell/JobRunner (.cursor/AGENTS.md). Install upstream with npx skills add ColinConwell/JobRunner --skill .cursor. Copyright stays with the author.

JobRunner -- Agent & Developer Guide

Project Overview

JobRunner is a Python library and CLI for configuration-driven job execution on remote compute. It targets SSH hosts, Google Cloud Compute Engine, and Hugging Face Jobs with a unified config format and modular backend/runner architecture.

Repository Layout

jobrunner/               # Main package (importable as `jobrunner`)
  __init__.py
  config/
    models.py            # Pydantic v2 models: HostConfig, JobConfig, SSHConfig, etc.
    loader.py            # YAML/TOML/Python config loading, env-var substitution, host merging
    defaults.py          # Built-in template host profiles (generic, no personal data)
  backends/
    base.py              # Abstract Backend class + registry (register_backend / get_backend)
    ssh.py               # Fabric-based SSH backend
    gcloud.py            # Google Cloud Compute Engine backend (uses gcloud CLI)
    huggingface.py       # Hugging Face Jobs backend (stub, extensible)
  runners/
    base.py              # Abstract Runner class + registry (register_runner / get_runner)
    script.py            # Plain Python script runner
    training.py          # DNN training runner (torchrun / JAX)
    docker.py            # Dockerized workload runner
    notebook.py          # Jupyter notebook executor (papermill / nbconvert)
    distributed.py       # Multi-node distributed training (torchrun / deepspeed)
  provisioners/
    base.py              # Abstract Provisioner + ProvisionerPipeline
    git.py               # Git clone/pull
    docker.py            # Docker image pull/build
    env.py               # Python env setup (venv/conda/uv)
    install.py           # System package installs (apt/pip/conda/uv)
  cli/
    main.py              # Typer app entrypoint (`jobrunner` command)
    commands/
      run.py             # `jobrunner run <config>`
      list_cmd.py        # `jobrunner list hosts|jobs`
      status.py          # `jobrunner status [host]`
      init.py            # `jobrunner init <output.yaml>`
  utils/
    ssh.py               # ~/.ssh/config parser, host auto-discovery
    logging.py           # Rich-based structured logging
    templates.py         # Jinja2 config templating
configs/
  hosts/                 # Template/example host profiles (committed to repo)
  jobs/                  # Template/example job configs (committed to repo)
devops/                  # Local host/job configs for real infrastructure (.gitignored)
  hosts/                 # Your real host profiles (YAML)
  jobs/                  # Your real job configs (YAML)
testing/                 # Test suite
  test_models.py         # Config model unit tests
  test_loader.py         # Config loader tests
  test_ssh.py            # SSH config parser tests
  local/                 # Integration tests (auto-discover configs from devops/)
scripts/
  setup_remote.sh        # Bootstrap script for new remote hosts
reference/               # Symlink to internal cloud-scripts repo (.gitignored)

Architecture Patterns

• Backend registry: Backends register themselves via register_backend("name", Class) at import time. The CLI and loader resolve backends by HostConfig.backend enum value.
• Runner registry: Same pattern -- register_runner("name", Class). Runners are resolved from RunnerConfig.type.
• Provisioner pipeline: ProvisionerPipeline runs a list of Provisioner instances in order (git -> install -> env -> docker). Each returns list[CommandResult].
• Config cascade: Job configs reference a host by name. The loader resolves the host from: (1) file-based profiles in devops/hosts/ or configs/hosts/, (2) built-in defaults in config/defaults.py, (3) auto-discovered SSH config entries. Docker configs merge host-level defaults with job-level overrides via deep merge.

Config File Conventions

• Template/example configs go in configs/ and are committed to the repo. These contain generic placeholder values and serve as starting points.
• Real/local configs go in devops/ and are gitignored. This is where developers put their actual host profiles and job configs with real hostnames, credentials, and project IDs.
• Job configs go in {devops,configs}/jobs/ (YAML preferred).
• Host profiles go in {devops,configs}/hosts/.
• Environment variables in config values use ${VAR} syntax and are substituted at load time.
• The config search path is: $JOBRUNNER_CONFIG_DIR > ./devops/ > ./configs/ > ./ > ~/.config/jobrunner/.

Setting Up Local Configs

1. Create devops/hosts/ and devops/jobs/ (already gitignored).
2. Copy a template from configs/hosts/ and fill in real values.
3. Integration tests in testing/local/ auto-discover configs from devops/.

Key Dependencies

• pydantic>=2.0 for config validation
• fabric>=3.0 for SSH execution (wraps paramiko)
• typer>=0.9 + rich for CLI
• pyyaml for YAML, tomllib (stdlib 3.11+) for TOML
• Optional: google-cloud-compute (gcloud extra), huggingface-hub (hf extra)

Development Commands (via Justfile)

• just install -- editable install with all extras
• just test -- run full test suite
• just test-unit -- unit tests only (no integration)
• just test-integration -- integration tests against real hosts
• just lint / just fmt -- linting and formatting
• just hosts -- list available hosts
• just run <config> -- dry-run a job config

Conventions

• Package source lives in jobrunner/ (flat, not under src/).
• Tests live in testing/.
• Local/real-infrastructure configs live in devops/ and are gitignored.
• Template/example configs live in configs/ and are committed.
• The reference/ directory is a symlink to an internal cloud-scripts repo and is gitignored.
• Config models use Pydantic v2 with model_validator for cross-field defaults.
• All backends and runners use a registration pattern -- new implementations should call register_backend() or register_runner() at module scope.
• Never commit personal hostnames, usernames, project IDs, or SSH key paths to the repo.

Adding a New Backend

1. Create jobrunner/backends/my_backend.py.
2. Subclass Backend, implement connect, disconnect, execute, upload, download.
3. Call register_backend("my_backend", MyBackend) at module scope.
4. Add the name to BackendType enum in config/models.py.
5. Import the module in cli/commands/run.py so it registers at CLI time.

Adding a New Runner

1. Create jobrunner/runners/my_runner.py.
2. Subclass Runner, implement run(backend, job, host) -> CommandResult.
3. Call register_runner("my_runner", MyRunner) at module scope.
4. Add the name to RunnerType enum in config/models.py.
5. Import the module in cli/commands/run.py.