cloud-pipeline

Imported from epam/cloud-pipeline (AGENTS.md). Install upstream with npx skills add epam/cloud-pipeline. Copyright stays with the author.

AGENTS.md

Repository conventions for AI coding agents. Any directory may carry its own AGENTS.md that adds detail for that subtree and takes precedence there; some are nested well below a module root. Check for one in every directory you change.

What this is

Cloud Pipeline wraps AWS/GCP/Azure compute and storage into a single platform for scientific computing: data processing pipelines, object/file storage management, Docker-based tool management, and GUI applications (noVNC/NoMachine/web endpoints) launched as containers on autoscaled cloud nodes.

A polyglot monorepo (Java + Spring Boot, React, Python 2/3, shell) built by a single root Gradle build. The version is declared in the root build.gradle.

Repository structure

Everything else is support: config/ (the checkstyle and pmd rulesets), docs/ (the mkdocs manual and release notes), gradle/ (the wrapper), jwt-generator/ (a CLI that mints API JWTs), vscode-cloud-pipeline/ (the "Cloud Pipeline Remote" VS Code extension), .github/ (the CI workflows, and Copilot's instructions) and .agents/, .claude/, .cursor/ (agent instructions and skills).

Names that mislead, or hide a coupling:

• api — its layering is a security boundary: authorization belongs in api/src/main/java/com/epam/pipeline/acl/, where 50 of the 54 annotated classes are. Four older *ApiService and manager classes still carry @PreAuthorize outside it; don't add a fifth. The Flyway migrations under api/src/main/resources/db/migration/ are append-only.
• core vs cloud-pipeline-common:model — both declare the same packages, and they have drifted apart, so the same class name in the two modules is often two different classes. No module depends on both, which is why nothing has ever collided.
• billing-report-agent — produces no reports. api queries the documents it writes and builds the reports from them.
• workflows/pipe-common — not a workflow. It is the pipeline Python package injected into every job container, so a break here breaks every run on the platform rather than one feature.
• scripts/ — not developer tooling despite the name: it ships in the distribution, and scripts/pipeline-launch/ runs inside every job container. A script supporting a development procedure belongs in that procedure's skill, under .agents/skills/<name>/scripts/.
• api/src/main/resources/static/ — a module copying an artifact here owes two more edits: the filename on the exclude list in client/build.gradle's buildUI, which deletes what is not listed, and the path in api.security.public.urls — in every file git grep -l api.security.public.urls names, not just one.
• deploy/docker/cp-edge — the nginx+Lua (OpenResty) reverse proxy that routes users into running job containers. It owns the proxy rules, but the endpoint URLs it serves are built in api (EdgeServiceManager, PipelineRunServiceUrlManager), so a routing change often needs both.
• data-transfer-service and data-sharing-service are unrelated despite the near-identical names.

Development process

A task that changes any file here — code, documentation or agent instructions — runs the implement-task skill (.agents/skills/implement-task/SKILL.md). Open that file before the first edit. Answering a question and reading code need none of it; reviewing someone else's diff may, to check what the change owed.

git add a new file as soon as you create it. Untracked, it is invisible to the checks, to your own reading of the diff, and to the commit.

.agents/localenv/ records how this workstation is set up — the version manager per language, the env names, where JAVA_HOME lives. It is gitignored and per-developer, so read it rather than assuming a toolchain, and never commit anything under it. If the directory does not exist, nobody has configured this machine yet: run the configure-dev-environment skill before any command that needs a toolchain. Attended only — unattended, the runner's toolchain belongs to the workflow, so just report what is missing.

A git worktree for this repository lives at .worktrees/<branch>, gitignored.

Files agents must not read

.env, .env.local and .env.<environment>.local files, secrets/, keystores and private keys, and the credential stores under ~ — .ssh, .aws, .azure, .config/gcloud, .config/gh, .kube/config, .gnupg. A .env.example is a template, not a secret, and is fine to read.

Anti-patterns

Things that have gone wrong here before, or that are expensive to undo:

• Don't weaken or skip a test to make a check pass. Fix the code, or report the change as blocked.
• Don't start a change without an issue number, and never invent one. Ask for it, or offer to file the issue — the implement-task skill → "Issue number".
• Don't carry on in the branch you were handed. A task gets its own branch, and whether to cut it from a branch that isn't shared — and whether to work in a worktree — are questions to ask, not defaults to assume — the implement-task skill → "Workspace".
• Don't skip the test because the code was already there. Every code change needs one — the implement-task skill → "Tests".
• Don't conclude that a change needs no documentation. An area with no page today is the gap itself, not the exemption — the implement-task skill → "Docs".
• Don't review your own diff. It goes to an agent that did not write it; where none can be spawned, review it as its own pass and say so — the implement-task skill → "Fresh-eyes review".
• Don't commit without reading the commit-changes skill.
• Don't edit an applied Flyway migration. Add a new one.
• Don't modernize a build file casually. compile/testCompile and bootRepackage are still in use, and a construct that looks obsolete may be load-bearing. Toolchain levels are per-module and mid-migration, so read that module's own build.gradle.
• Don't run deployment tooling (pipectl, docker push, deploy/pipectl/build-pipectl.sh, aws s3 rm|mv) as part of a code change.
• Don't document what you did not verify. Prefer a command that finds the answer over a copied value that will drift.
• Don't claim a command passed if you did not run it. Say what you actually ran.