jira-to-openproject

Imported from netresearch/jira-to-openproject (AGENTS.md). Install upstream with npx skills add netresearch/jira-to-openproject. Copyright stays with the author.

AGENTS.md (root)

Overview

j2o is a Python 3.14 migration toolset that moves project management data from Jira Server 9.x to OpenProject 17.3+ — users, projects, work packages, custom fields, statuses, workflows, attachments, Tempo time logs, agile boards, and reporting artefacts. It is a 41-module ETL pipeline with a FastAPI dashboard.

Precedence: The closest AGENTS.md to changed files wins. Root holds global defaults only.

Global rules

• Keep PRs small (~≤300 net LOC), land tests with code
• Conventional Commits: feat(scope):, fix:, docs:, refactor:
• Ask before: heavy deps, full e2e suites, repo-wide rewrites
• Never commit secrets or PII; use .env.local for overrides

Commands (verified 2026-02-09)

Setup

cp .env.example .env        # Configure credentials
uv sync --frozen            # Install dependencies into .venv
.venv/bin/pytest -x         # Verify setup

Prerequisites: Python 3.14+, uv, Docker (for container tests), SSH access to the OpenProject host.

Development

• Create a feature branch: git switch -c feat/<scope>.
• Iterate locally with make dev-test (fast path) before running make container-test.
• Lint via make lint; format via make format.
• Commit with Conventional Commits (feat(scope):, fix:, docs:, etc.).

Architecture

See docs/ARCHITECTURE.md for the full layer breakdown. Short form:

Jira API ─▶ src/infrastructure/jira/jira_client.py ─┐
                                        ├─▶ src/migrations/*.py (ETL) ─▶ src/infrastructure/openproject/openproject_client.py ─▶ OpenProject API / Rails console
Tempo API ─▶ src/clients/tempo_client.py ┘

Every migration inherits BaseMigration and implements _extract → _map → _load (see the Golden samples section). Cross-cutting concerns — retries, checkpointing, mappings, provenance — live in src/utils/ and src/mappings/.

Testing

Golden sample: tests/unit/test_config_loader_security_enhanced.py. New migration modules land with matching unit tests in tests/unit/ and, when applicable, functional tests in tests/functional/.

File Map

src/               → Core Python package (clients, migrations, mappings, models, utils)
src/infrastructure/ → Jira/OpenProject/SSH/Docker/RailsConsole adapters (jira/, openproject/)
src/migrations/    → 41 extract→map→load migration modules
src/dashboard/     → FastAPI admin dashboard (Vue + Chart.js + WebSocket)
src/ruby/          → Ruby template scripts for Rails console execution
src/utils/         → Shared helpers (retry, checkpoint, timezone, markdown)
tests/             → Pytest suites: 79 unit, 17 functional, 16 integration
scripts/           → Operational helpers (tmux bootstrap, rehearsal, migration shells)
config/            → YAML configuration, schemas, environment templates
docs/              → Architecture, entity mapping, developer guide, security
contrib/           → OpenProject .irbrc and contributed assets
var/               → Runtime data, logs, caches, checkpoints (gitignored)

Golden samples

Heuristics

Critical constraints

• Python 3.14 PEP 649 — NEVER enable ruff's TC001/TC002/TC003; annotations are evaluated eagerly and TYPE_CHECKING-only imports break at runtime.
• Mappings proxy — BaseMigration.__init__ must use self.mappings = config.mappings (proxy), not config.get_mappings(); tests monkeypatch the proxy.
• Never commit secrets — credentials belong in .env.local; verify via git diff before staging.
• Never work on main — always land changes through feature branches and PRs.

Lint & type conventions

• mypy scope — files = ["src"]; tests/ and scripts/ are excluded by design (tests duck-type internals; scripts are one-off). A per-module override on JSON-API-boundary modules (src.infrastructure.jira.*, src.infrastructure.openproject.*, src.application.components.*, src.utils.*, src.dashboard.*, src.migration, src.main) disables structural codes (attr-defined, arg-type, union-attr, index, return-value, … the dict-shape family) because Jira/OP JSON can't be statically proven. Name/import codes still fire everywhere. A new src/ module that handles typed data (pydantic/internal helpers) must NOT join the override — let the codes fire. Tighter typing path = add TypedDict/pydantic at the boundary, then re-enable codes.
• Never put "str"/"int"/"object" in disable_error_code — not valid mypy codes; mypy crashes with KeyError on startup. Use str-bytes-safe/str-format if that's what you mean.
• ruff — select = ["ALL"]. Non-production dirs carry per-file-ignore blocks (scripts/**, examples/**, config/**, check_*.py, jira/** local stub). scripts/** also ignores F821 (scripts embed Ruby in f-strings where #{var} isn't a Python name). When a new rule fires only in those dirs, extend the matching per-file block — don't add to the project-wide ignore unless src/ needs it too. D203/D211, D212/D213, COM812 stay in the global ignore (mutually incompatible / conflicts with the formatter).

CI merge gate & SonarCloud

• Required checks (branch protection — these block merge): container-tests, Tests (pytest), Lint (ruff), Type check (mypy), Python Dependency Audit, gitleaks. The copilot_code_review ruleset also requires a Copilot review on the latest commit (re-request after every force-push).
• SonarCloud is advisory, not required — a red SonarCloud check does NOT block merge (mergeStateStatus shows UNSTABLE, still mergeable). Its new-code Quality Gate fails on security hotspots, worth clearing up front to avoid a CI round-trip + Copilot re-review:
  • Prefer https:// over http:// in test fixtures where the scheme isn't the thing under test — clear-text URLs are flagged as a hotspot (S5332), which is not ignored for tests/. (When the scheme genuinely is under test, the http:// is expected.)
  • /tmp literals — python:S5443 is already ignored for tests/** in sonar-project.properties, so test /tmp paths don't fail the gate; production /tmp usage is a reviewed hotspot. To also silence the github-advanced-security alert on a test path, follow the existing _CONTAINER_TMP = "/" + "tmp" concatenation pattern (tests/unit/test_bulk_create_cleanup_user.py), which keeps precise assertions while dodging Sonar's static matcher.

Architecture contract (import-linter)

.importlinter enforces a Cosmic Python layered contract (CI job architecture runs uv run lint-imports). Layer ordering, highest to lowest:

src.dashboard | src.main  >  src.application  >  src.infrastructure  >  src.domain : src.models

Cross-cutting modules are intentionally outside the contract and may be imported from any layer: src.utils, src.config, src.mappings, src.display, src.migration. Preserve this layering on any structural change.

Migration fidelity gotchas

• Never raise from JiraIssueService._fetch_single_chunk. BatchProcessor.process_batches (reached via PerformanceOptimizer.batch_processor.process_batches) runs each batch in a thread pool and catches per-future exceptions by dropping the whole batch (≈100 keys) — raising to "fail fast" silently amplifies data loss without aborting. Recover in place (e.g. classify by HTTP status, halve+retry on 413/414) or log+continue.
• Work-package date flow — the default profile is work_packages_skeleton (sets work_packages.created_at/updated_at from Jira via the bulk-create Ruby update_columns) + work_packages_content (comment journals via journal_notes/journal_user + save!, which stamps NOW). The WP "Created on" is preserved by the skeleton step; comment/journal dates must be back-dated by carrying comment.created through and running update_columns(created_at:) after save!. Do NOT rewrite the journal validity_period (tstzrange with a per-journable non-overlap exclusion constraint) when back-dating — update_columns(created_at:) alone drives the displayed date.

Boundaries

Always: Check git status first · Use feature branches · Run make dev-test before commit Ask first: Heavy deps · Repo-wide rewrites · Full e2e suites · Deleting migration data Never: Commit secrets · Work on main · Skip tests · Hardcode credentials

Codebase state

• Python 3.14+, uv package manager, Docker Compose test profile
• Target: Jira Server 9.x → OpenProject 17.3+ (currently tested against 17.3)
• OpenProject 15+ requires journal_notes/journal_user + save! pattern (not journals.create!)
• J2O_FORCE_RAILS_RUNNER=1 bypasses tmux console for rails runner mode
• 41 migration modules, 307 projects migrated, 65K+ work packages
• Rails runner timeout: 300s (increased from 120s for large projects like ADKP)
• Watcher migration uses batch insert_all with pre-fetched ID sets

Terminology

Scope index

Key documentation

• docs/ENTITY_MAPPING.md — Jira→OpenProject field mappings
• docs/MIGRATION_COMPONENTS.md — Module catalog with dev state
• docs/ARCHITECTURE.md — Client layer design
• docs/DEVELOPER_GUIDE.md — Development standards
• docs/SECURITY.md — Security practices

Quick start

cp .env.example .env        # Configure credentials
uv sync --frozen            # Install dependencies
make dev-test               # Verify setup (run unit tests)
make start-rails            # Start tmux Rails console
j2o --help                  # Discover CLI commands

When instructions conflict

Nearest AGENTS.md wins. User prompts override files.