Imported from norfablabs/NORFAB (
AGENTS.md). Install upstream withnpx skills add norfablabs/NORFAB. Copyright stays with the author.
AGENTS.md - NorFab Repository Guide
Project Overview
NorFab (Network Automations Fabric) is a Service-Oriented Architecture (SOA) framework for extreme network automation. It runs equally on Windows, Linux, and macOS — locally on a laptop or distributed across servers.
- License: Apache-2.0
- Python: 3.10–3.14
- Docs: https://docs.norfablabs.com
- Repo: https://github.com/norfablabs/NORFAB
Architecture
Three core components communicate via ZeroMQ using the NFP (NorFab Protocol):
Clients ──► Broker ──► Workers
- Broker (
norfab/core/broker.py) — Central message router. Distributes jobs from clients to available service workers. - Workers (
norfab/workers/) — Service processes that execute automation tasks. Multiple workers can form a service (load-balanced). - Clients (
norfab/clients/) — Interfaces to submit jobs and retrieve results (Python API, CLI, Robot Framework). - Inventory (
norfab/core/inventory.py) — YAML-based configuration loaded at startup; supports glob patterns for worker config mapping.
NFP Protocol
The communication protocol is defined in norfab/core/NFP.py. Workers, broker, and clients speak NFP over ZeroMQ sockets. Jobs are tracked by UUID in a SQLite database on the client side (norfab/core/client.py).
Job Lifecycle
NEW → SUBMITTING → DISPATCHED → STARTED → COMPLETED / FAILED / STALE
Directory Structure
norfab/
├── core/
│ ├── nfapi.py # NorFab main class — starts broker + workers + client
│ ├── broker.py # NFPBroker — routes jobs between clients and workers
│ ├── worker.py # NFPWorker base class — all service workers extend this
│ ├── client.py # NFPClient — submits jobs, stores results in SQLite
│ ├── inventory.py # NorFabInventory — loads YAML inventory files
│ ├── NFP.py # Protocol constants and message builders
│ ├── keepalives.py # Keepalive heartbeat implementation
│ ├── security.py # ZeroMQ certificate generation
│ └── exceptions.py # Custom exceptions
├── workers/
│ ├── nornir_worker/ # Nornir network automation service
│ ├── netbox_worker/ # NetBox DCIM/IPAM integration service
│ ├── agent_worker/ # AI/LLM agent service (LangChain/Ollama)
│ ├── fastapi_worker/ # REST API service (FastAPI + Uvicorn)
│ ├── fastmcp_worker/ # Model Context Protocol (MCP) service
│ ├── workflow_worker/ # Workflow orchestration service
│ ├── containerlab_worker/ # ContainerLab integration
│ └── filesharing_worker/ # File sharing service
├── clients/
│ ├── nfcli_shell/nfcli_shell_client.py # Interactive CLI (nfcli)
│ ├── robot_client.py # Robot Framework library
│ ├── textual_client.py # TUI client (Textual)
│ └── nfweb/ # Local web client; topology is its first application
├── models/
│ ├── norfab_configuration.py # Pydantic models for inventory config
│ ├── norfab_configuration_logging.py # Logging config models
│ ├── fastapi/ # FastAPI response models
│ └── containerlab/ # ContainerLab models
└── utils/
└── nfcli.py # CLI entry point script
tests/
├── conftest.py # pytest fixtures (NorFab start/teardown)
├── nf_tests_inventory/ # Test inventory (inventory.yaml + service configs)
├── services/
│ ├── containerlab/ # Containerlab service tests by task area
│ ├── dummy/ # Dummy plugin service tests
│ ├── fakenos/ # FakeNOS service tests by task area
│ ├── fastapi/ # FastAPI service tests by task area
│ ├── fastmcp/ # FastMCP service tests by task area
│ ├── filesharing/ # FileSharing service tests by task area
│ ├── netbox/ # NetBox service tests by task area plus common.py helpers
│ ├── nornir/ # Nornir service tests by task area
│ └── workflow/ # Workflow service tests by task area
├── clients/
│ └── nfweb/ # NFWeb API, storage, and collector tests
└── nfcli/ # Interactive CLI shell tests
├── test_shell_client.py
└── test_shell_common.py
docs/ # MkDocs documentation source (Material theme)
docker/ # Docker deployment configs
Common Commands
Installation
# Core only
poetry run pip install norfab
# With CLI
poetry run pip install norfab[nfcli]
# With Nornir service
poetry run pip install norfab[nornirservice]
# Everything
poetry run pip install norfab[full]
# Development (using Poetry)
poetry install -E docs
Developer Task Automation
Use Invoke from the repository root for common development tasks:
poetry run inv --list
poetry run inv checks
poetry run inv docs-build
poetry run inv docs-serve
poetry run inv docker-tests-core
poetry run inv docker-tests-nornir
Docker suite tasks use the canonical docker-tests-<suite> form and also
accept docker-test-<suite> aliases. Use task help to see selectors, marker,
keyword, Python-version, build, and file-parallel options:
poetry run inv --help docker-tests-nornir
Running
# Start interactive CLI (from directory containing inventory.yaml)
poetry run nfcli
# Create a new NorFab environment scaffold
poetry run nfcli --create-env norfab
Testing
# Run pytest suites in isolated Docker runners
poetry run inv docker-tests-core
poetry run inv docker-tests-nornir
poetry run inv docker-tests-netbox
# Run one test or select a marker subset
poetry run inv docker-tests-nornir --selector=tests/services/nornir/test_worker.py
poetry run inv docker-tests-netbox --marker="netbox and netbox_get_devices"
# Discover test_*.py and run one isolated container per file, two at a time
poetry run inv docker-tests-netbox --parallel-runs=2
# Validate/build all Docker test services or run the distributed topology
poetry run inv docker-tests-config
poetry run inv docker-tests-build
poetry run inv docker-tests-distributed
Docker runtime files and JUnit reports are stored under the selected service's
ignored docker/norfab-docker-tests/<service>/__norfab__/ directory.
Docker suite tasks scope pytest collection to the suite's conventional test
directory before applying its marker; an explicit --selector overrides that
default collection root.
The NetBox suite is further split into test-file Invoke runners named
docker-tests-netbox-<file>, such as docker-tests-netbox-crud. Each uses
docker compose run with the shared
NetBox test image, an explicit norfab-tests-netbox-<group>-<run-id> container
name, and a file-specific runtime/JUnit directory. docker-tests-netbox and
docker-tests-all launch all of these dedicated group containers.
With --parallel-runs=N, test roots are derived from tests/services/<suite>,
tests/clients/<suite>, or tests/<suite> and per-file runtimes are stored
under <service>/parallel/<test-file>/__norfab__/. At most N per-file
containers run concurrently.
Individual docker-tests-<suite> tasks report the container exit status but
do not fail Invoke; docker-tests-all runs the regular suite containers in
parallel and returns non-zero after summarizing all failed suites. It excludes
the Containerlab suite, idle performance profiler, and distributed topology,
which remain available through their dedicated tasks.
Every Docker suite invocation writes a timestamped Markdown summary under
docker/norfab-docker-tests/reports/, including runs narrowed by selectors,
markers, keywords, extra pytest arguments, or per-file parallelism. Reports are
based only on JUnit XML artifacts created or updated by that invocation;
docker-tests-all produces one consolidated report for the complete run.
The distributed task validates the client's cached broker public certificate;
if an older runtime key is present, rerun it with --force-certificates to
replace that public certificate. It never copies the broker private key.
Direct pytest remains available for focused local debugging:
# Run all tests (from repo root, requires a running/startable NorFab)
cd tests && poetry run pytest
# Run a specific service test suite
cd tests && poetry run pytest services/nornir
# Run NFCLI shell tests
cd tests && poetry run pytest nfcli
cd tests && poetry run pytest -m nfcli
# Run tests with output
cd tests && poetry run pytest -s -v
Linting & Formatting
# Run all non-mutating checks (Black, Ruff, and Vulture)
poetry run inv checks
# Format with Black
poetry run inv format
# Lint with Ruff
poetry run inv lint
# Ruff auto-fix
poetry run ruff check . --fix
# Report dead code; findings are not suppressed or auto-fixed
poetry run inv dead-code
Important formatting Rules
- Workers
job.eventcall messages must start with lowercase letters;job.eventcalls support setting event severity throughseverity=WARNING/INFO/ERROR - Logging calls, e.g.
log.info, must start with lowercase letters - Any spelling mistakes in docstrings, comments or variable names must be fixed
Documentation
# Serve docs locally
poetry run inv docs-serve
# Build docs
poetry run inv docs-build
Feature Documentation Maintenance
docs/norfab_features.mdis the evaluator- and RFP-oriented catalogue of NORFAB capabilities.- When adding, changing, deprecating, or removing a code feature, review and update the relevant feature wording on this page in the same change.
- Keep each entry concise and verify its description, supported interfaces, use cases, limitations, documentation links, and the page's Last updated date.
Inventory File Structure
NorFab is configured via a YAML inventory.yaml. The default search path is ./inventory.yaml.
broker:
endpoint: "tcp://127.0.0.1:5555"
logging:
handlers:
terminal:
level: CRITICAL
file:
level: DEBUG
workers:
nornir-*: # glob pattern — applies to all matching workers
- nornir/common.yaml
nornir-worker-1: # specific worker name
- nornir/nornir-worker-1.yaml
topology:
broker: True # start broker in this process
workers:
- nornir-worker-1 # start these workers in this process
Worker config files are merged recursively — glob patterns first, then specific names.
Python API Usage
from norfab.core.nfapi import NorFab
# Start NorFab (broker + workers + client)
nf = NorFab(inventory="./inventory.yaml")
nf.start()
client = nf.make_client()
# Run a job
result = client.run_job(
service="nornir",
task="cli",
workers="nornir-worker-1",
kwargs={"commands": ["show version"]}
)
nf.destroy()
Worker Plugin System
Workers are registered via Python entry points in pyproject.toml:
[project.entry-points."norfab.workers"]
"nornir" = "norfab.workers.nornir_worker.nornir_worker:NornirWorker"
"netbox" = "norfab.workers.netbox_worker.netbox_worker:NetboxWorker"
"fastapi" = "norfab.workers.fastapi_worker.fastapi_worker:FastAPIWorker"
"agent" = "norfab.workers.agent_worker.agent_worker:AgentWorker"
"workflow" = "norfab.workers.workflow_worker.workflow_worker:WorkflowWorker"
"containerlab"= "norfab.workers.containerlab_worker.containerlab_worker:ContainerlabWorker"
"fastmcp" = "norfab.workers.fastmcp_worker.fastmcp_worker:FastMCPWorker"
"filesharing" = "norfab.workers.filesharing_worker.filesharing_worker:FileSharingWorker"
"fakenos" = "norfab.workers.fakenos_worker.fakenos_worker:FakeNOSWorker"
Custom workers can be registered by installing a package that declares the same entry point group. Set service: <name> in the worker's inventory config.
Key Design Patterns
- All workers extend
NFPWorker(norfab/core/worker.py). Task methods decorated with@taskare auto-registered and callable by clients. - Pydantic models are used for all job inputs/output, API validation, and documentation generation.
- Jinja2 templating is supported inside inventory YAML files.
- SQLite stores client-side job results and events (
ClientJobDatabaseinclient.py). Database files are stored in__norfab__/directories (gitignored). - ZeroMQ with optional CurveZMQ encryption (
security.py) handles all inter-process communication. - Multiprocessing — broker and each worker run in separate OS processes (
multiprocessing.Process).
Ruff Lint Rules
Configured in pyproject.toml. Ignored rules:
E712— true-false-comparisonE501— line too longANN401— disallowAnyannotation
Selected rule sets: E, F, I, ANN
Gitignored Patterns
__norfab__/directories (runtime data: job DBs, certs, logs)private/directorysite/(built docs)*.log,*.pyc,dist/,build/
Community & Support
- Slack: Networktocode
#norfabchannel / NetDev Community - GitHub Discussions: https://github.com/norfablabs/NORFAB/discussions
References
- NFWeb developer guide:
docs/development/nfweb_developer_guide.md - Task Pydantic models:
docs/development/tasks_pydantic_models_guide.md - Documentation style guide for docs changes:
docs/development/documentation_style_guide.md - Feature catalogue:
docs/norfab_features.md - Testing framework:
docs/testing/norfab_testing_framework.md - NetBox service tests and refactoring guidance:
docs/testing/netbox_service_tests.md - Invoke developer automation ADR:
docs/development/adr_invoke_developer_automation.md - Docker test commands:
docker/norfab-docker-tests/README.md
