Imported from thomassuedbroecker/galaxium-travels-watsonx-orchestrate-mcp-bob-guide (.bob/skills/watsonx-orchestrate/SKILL.md). Install upstream with npx skills add thomassuedbroecker/galaxium-travels-watsonx-orchestrate-mcp-bob-guide --skill watsonx-orchestrate. Copyright stays with the author.

IBM watsonx Orchestrate (wxO) — Build · Test · Debug · Publish

Authoritative, end-to-end guide for delivering production agents on IBM watsonx Orchestrate with the Agent Development Kit (ADK). It is grounded in the real ADK source and CLI (ibm-watsonx-orchestrate), not guesswork.

Golden rule: the ADK changes fast. The CLI specifics here were live-verified against ibm-watsonx-orchestrate 2.15.0 (IBM Cloud SaaS, us-south, 2026-08-18). When a command, flag, or YAML field is uncertain — or you're on a different version — verify against the live tool with orchestrate <group> --help and orchestrate <group> <cmd> --help before running it. Treat third-party blog syntax as approximate; trust --help and this skill's references. Check your version with orchestrate --version / pip show ibm-watsonx-orchestrate. Upgrade with pip install -U ibm-watsonx-orchestrate (needs Python ≥3.11, <3.15).

What's new in 2.14 → 2.15 (live-verified on SaaS unless marked otherwise — evidence in test/house_clinic/): orchestrate controls binds policy artifacts (PII filter, guardrails, secrets detector, rate limiter, …) to agents/tools/models at execution hooks — the flagship 2.15 feature, and it really fires at runtime (§4a); an agent may now reference multiple knowledge bases, and each one shows up as its own retrieval tool (§3.3); welcome_content gains is_user_barge_in_disabled (§3.3); voice configs gain Google TTS, Deepgram normalize_volume, and finer idle control (use_llm_generated_idle_message, repeat_previous_message, long_running_task_seconds) (§4b); connections configure accepts --name to override the api_key header (§5); every document-processing node accepts language=, and the classifier is capped at 30 classes (§3.2/agents-tools ref); flows gain on_flow_abort / on_flow_delete callbacks; external agents can now export traces to the Analytics dashboard via OpenTelemetry or the Observability SDK (§6b); new model provider redhat-ai and external-agent provider msftstudio (§4).

⚠ Three release-note claims do NOT match the live 2.15.0 build. They are named exactly as the release note words them, so you can recognize them: the welcome-message cap is documented as "100 → 1000 characters" but the API still hard-rejects anything over 100 (422); the provider is documented as red_hat_ai but the enum value is redhat-ai; the external-agent provider is documented as microsoft_copilot_studio but the enum value is msftstudio. Use the enum values.

Critical

• Always run the script source ./.bob/skills/watsonx-orchestrate/references/setup-venv.sh before doing anything else and before running the orchestrate CLI!
• After code generation always deploy/import and runs tests (see 3.5 Test)

1. Mental model — what you are building

watsonx Orchestrate runs agents that route user requests to tools, collaborator agents, and knowledge bases, powered by an LLM.

The two surfaces you work through:

• The orchestrate CLI — manages every resource in the active environment.
• The Developer Edition — a full local wxO that runs on Docker (orchestrate server start) so you can iterate before touching production.

2. Prerequisites

Network access to your wxO instance (SaaS or on-prem) — or Docker for the local Developer Edition is required. Important: Always run the script source ./.bob/skills/watsonx-orchestrate/references/setup-venv.sh before doing anything else! The setup-venv.sh creates a virtual environment with Python and the orchestrate CLI.

Everything below works against whatever environment is active. The same agent/tool artifacts deploy unchanged to local, SaaS, or on-prem — only the environment you activate and the connection credentials differ.

2a. Connect to your environment (do this first)

orchestrate runs every command against the active environment. Pick one of three targets; SaaS and on-prem are the production paths, local is optional.

SaaS (IBM Cloud) — verified working flow

Use the API service URL (the one containing /instances/<id>), not the console URL, and your IBM Cloud API key.

orchestrate env add -n my-saas \
  -u https://api.<region>.watson-orchestrate.cloud.ibm.com/instances/<INSTANCE_ID>
orchestrate env activate my-saas --api-key "$IBM_CLOUD_API_KEY"
orchestrate agents list          # confirm you're connected

Auth type is auto-inferred (typically ibm_iam for IBM Cloud SaaS). If inference is wrong, set it explicitly on env add with --type [ibm_iam|mcsp|mcsp_v1|mcsp_v2|cpd].

On-prem (Cloud Pak for Data / CPD)

orchestrate env add -n my-onprem -u https://<cpd-host>/orchestrate --type cpd \
  [--insecure | --verify /path/to/ca.crt]      # for self-signed / private CAs
# Authenticate with an API key OR username+password (not both):
orchestrate env activate my-onprem --api-key "$CPD_API_KEY"
orchestrate env activate my-onprem -u "$CPD_USER" -p "$CPD_PASSWORD"

Local Developer Edition (optional — offline iteration)

A full local wxO on Docker (Rancher/Colima): 16 GB RAM / 8 cores / 25 GB disk (≥19 GB if Document Processing -d). Needs an entitlement key in .env.

orchestrate server start -e .env --accept-terms-and-conditions
orchestrate env activate local
orchestrate chat start            # local-only chat UI

⚠ Excel input to the Document Extractor is off by default in Developer Edition 2.15.0. The node supports .xlsx, but you must opt in — add to .env and restart the server:

GLOBAL_WORKER_DOCLING_PIPELINE=true

Known limitation while that is on: only one document at a time. Uploading or ingesting several concurrently — via Document Processing tools or knowledge ingestion — can fail. Plan demos accordingly; IBM say this is temporary.

Environment hygiene

orchestrate env list                 # the active one is starred; that's where imports land
orchestrate env activate <name>      # switch targets
orchestrate env remove --name <name>

Always confirm the active env before importing — the #1 cause of confusion. Keep secrets in a gitignored .env and pass via "$VAR" so keys stay out of shell history. Treat the IBM Cloud API key like a password; rotate if exposed.

See references/cli-reference.md for the full command catalog. Need example projects? The public ADK repo's examples/ directory and the official docs are the canonical source — see §10.

3. The canonical lifecycle

Follow this order. Dependencies must exist before the thing that references them.

setup `orchestrate` CLI → scaffold project → write tools (+connections/models/KB/skills) 
   → write agent YAML → run `import-all.sh` (1. import connections → 
   2. import models → 3. import KB → 4. import tools/toolkits → 5. import skills → 
   6. import agent) → run test (see 3.5 Test) → debug → re-import → deploy to production

3.1 Scaffold

my_agent/
├── README.md
├── agents/            my_agent.yaml
├── tools/             *.py  (one @flow per file; @tool can be grouped)
├── connections/       *.yaml (kind: connection)
├── knowledge_base/    *.yaml + source docs
├── skills/            <skill-name>/SKILL.md (+ optional WXO.yaml) — 2.13.0 agent skills
├── models/            *.yaml (kind: model) — only if adding a custom model
├── import-all.sh      orchestrate ... import commands, dependency-ordered
├── delete-all.sh      orchestrate ... delete commands, counter-part to import-all.sh
└── .env               secrets (gitignored)

3.2 Write a Python tool

from ibm_watsonx_orchestrate.agent_builder.tools import tool, ToolPermission
from pydantic import BaseModel

class WeatherInformation(BaseModel):
    city: str
    temp_c: float

@tool(permission=ToolPermission.READ_ONLY)   # or READ_WRITE
def get_weather(city: str) -> WeatherInformation:
    """
    Get the current weather for a city.

    Args:
        city (str): Name of the city to look up.
    Returns:
        WeatherInformation: Temperature in celsius for specific city.
    """
    return WeatherInformation(
        city=city,
        temp_c=21.1)

Decorator rules and docstring grammar are strict — getting them wrong is the most common import failure. See §5 and references/agents-tools-schemas.md.

⚠ You will see this on EVERY Python tool import — it is a false positive. Do not "fix" it.

[WARNING] - Unable to properly parse parameter descriptions due to missing or incorrect
            type hints. This may result in degraded agent performance.

It fires on correctly-annotated tools and the descriptions still parse fine (live-verified 2.12.0 and 2.13.0). Chasing it wastes a lot of time and often makes the tool worse. Only act if the parsed schema is actually missing descriptions; the real causes are a genuinely missing type hint, or a blank line between the Args: and Returns: blocks.

3.3 Write the agent YAML

spec_version: v1
kind: native
name: weather_agent
description: Returns weather information for a location.
instructions: >
  You are a helpful weather assistant. When the user asks about weather,
  call the get_weather tool with the city name and present the result clearly.
llm: watsonx/meta-llama/llama-3-3-70b-instruct   # example only — pick one that EXISTS in `orchestrate models list` (§4)
style: react_core                                 # 2.13.0 DEFAULT; `default`, `react`, `planner` are DEPRECATED (omit to take the default)
tools:
  - get_weather
starter_prompts:
  prompts:
    - id: default0
      title: Check weather
      subtitle: Ask about any city        # optional — shown below the title
      prompt: What's the weather in Boston?
welcome_content:
  welcome_message: Welcome to the Weather Agent      # ⚠ HARD 100-character limit (below)
  description: Ask me about the weather in any city.
  is_user_barge_in_disabled: false   # 2.15.0 — true = the user cannot interrupt the
                                     #   agent's spoken opening message (voice agents)
knowledge_base:                      # 2.15.0 — an agent may reference MORE THAN ONE
  - city_climate_notes
  - regional_weather_policy
# ⚠ starter_prompts: do NOT add `is_default_prompts:` or `state:` — they are
#   silently ignored by the platform and cause prompts not to render in the UI.
#   Valid fields per prompt: id, title, subtitle, prompt. (live-verified 2.12.0)

⚠ welcome_message is still capped at 100 characters (live-verified 2.15.0). The 2.15.0 release note says the cap moved from 100 to 1000, but that changed only the recommendation in the docs — the API still rejects a longer message outright:

Failed to create agent: [{'type': 'value_error',
  'loc': ['body','additional_properties','welcome_content','welcome_message'],
  'msg': 'Value error, At most 100 characters are allowed for Welcome message.'}]

Put the long version in description: (uncapped in practice) and keep welcome_message short.

Multiple knowledge bases (2.15.0) — how routing actually works

Listing several KBs is not "one bigger corpus". Each knowledge base is exposed to the agent as its own retrieval tool, named after the KB, and the router picks between them the same way it picks any tool — from the KB's description. Live-verified: a single question spanning two KBs produced two separate tool calls in step_history (["ppth_formulary", "ppth_policy"]) and two matching observations in the trace.

So, when splitting a corpus across KBs:

• Write each KB description as a routing trigger ("drugs, dosing, what is restricted"), not a summary. Make them mutually exclusive.
• Say in instructions: which question goes to which KB, and that a question spanning both must consult both — otherwise the agent will often stop after the first hit.
• Splitting is worth it when the sub-corpora answer different kinds of question. One homogeneous corpus should stay one KB; two KBs competing for the same trigger is a coin flip.

⚠ Built-in KB ingestion rejects .md. Unsupported file type text/markdown — despite agent skills being authored as SKILL.md. Supported: PDF/DOCX/PPTX/XLSX/CSV/ HTML/TXT. Rename .md → .txt (live-verified 2.15.0).

Required fields: spec_version, kind, name, description. Strongly recommended: instructions, llm, style, tools, plus starter_prompts / welcome_content for UX. Full field reference (collaborators, knowledge_base, guidelines, structured_output, chat_with_docs, voice, channels) is in references/agents-tools-schemas.md.

3.3a Production-grade agent fields (ADK 2.11+, verify before relying on)

For long-running or recurring agents, add these optional blocks (sourced from the ADK docs — confirm with orchestrate agents create --help / the schema reference):

# Conversation compaction — prevents context overflow in long chats
compaction_settings:
  context_compaction_enabled: true
  context_compaction_threshold: 20000   # tokens that trigger Level-1 compaction
  compaction_sliding_window: 10         # most-recent messages kept verbatim
  large_message_threshold: 50000        # token size that marks a "large" message
  large_message_chunk_size: 30000
  large_message_target_summary: 10000
  large_message_detect_structured: true

# Per-agent LLM decoding params (greedy/temperature, limits, response format, …)
llm_config:
  temperature: 0
  max_tokens: 2048
  # top_p / top_k / seed / response_format / reasoning_effort / provider extensions

is_schedulable: true     # enables recurring runs; internal scheduling tools auto-appear
# restrictions: …        # access restrictions (see schema reference)

3.3b Orchestrator / multi-agent (collaborators) — live-verified 2.13.0

A native agent becomes an orchestrator (supervisor) that uses other agents as agents by listing them under collaborators:. wxO's multi-agent pattern:

name: dr_house
style: react_core               # 2.13.0 default; supports collaborators (experimental_customer_care does NOT)
tools:                          # the orchestrator can still call its own tools
  - differential_diagnosis
collaborators:                  # other agents, by snake_case name — imported FIRST
  - dr_wilson                   # oncology specialist agent
  - dr_cuddy                    # administration agent
  - dr_foreman                  # neurology agent
skills:                         # 2.13.0 — attach agent skills by name (kebab-case!) — see §3.3c
  - diagnostics-protocol

How it works (verified end-to-end):

• Import/deploy collaborators before the orchestrator (they're dependencies). Deploy all of them on SaaS so runtime delegation can reach them.

• At runtime wxO auto-generates a chat_with_collaborator_<name> tool on the orchestrator. The router picks a collaborator the same way it picks a tool — from the collaborator's description. So collaborator descriptions must be distinct and routing-friendly (lead with the specialty/trigger). The orchestrator then frames the collaborator's reply in its own voice.

• Delegation is visible in the run step_history/reasoning trace as a tool_calls step named chat_with_collaborator_<name>.

  ⚠ The specialty is appended to the name — inconsistently. Match by PREFIX, never by equality. Live-verified 2.13.0 in a single run set: dr_wilson → chat_with_collaborator_dr_wilson_oncology, dr_cuddy → chat_with_collaborator_dr_cuddy_administration, but dr_chase → chat_with_collaborator_dr_chase (no suffix). An assertion written as name == "chat_with_collaborator_dr_wilson" reports a false failure on a run that routed perfectly. Use name == f"chat_with_collaborator_{agent}" or name.startswith(f"chat_with_collaborator_{agent}_").

• Collaborators are themselves full agents (own tools/KB/collaborators) — nesting works: live-verified a 3-level chain House → Foreman → Chase where the mid-tier agent (Foreman) used its own tool AND sub-delegated. The whole nested chain (sub-agent tool calls + sub-delegations) is flattened into the top-level run's step_history.

• An agent cannot list itself as a collaborator (circular reference rejected).

Routing tip: the router picks a collaborator from its description, so write that field as a routing trigger — lead with the specialty and the concrete findings/keywords that should land there — not as a personality blurb. Make the descriptions mutually exclusive; two agents competing for the same trigger is a coin flip.

If the orchestrator answers instead of delegating, look at its OWN instructions first. A single permissive sentence ("do not delegate a question you can answer with your own tools") suppresses routing entirely — live-verified, removing one such sentence moved routing accuracy from 5/9 to 8/9 with no other change. Make delegation the stated default, add mandatory rules for domains the orchestrator's own tools could plausibly cover, and give the orchestrator as few tools as possible — every tool it owns is a reason not to route. Full playbook: references/testing-debugging.md §2a.

⚠ Scheduling caveats (live-verified 2.12.0):

• is_schedulable: true set in YAML did not persist — import succeeded but the round-trip export came back is_schedulable: false. Scheduling needs to be enabled at the tenant level first (per the platform "enable scheduling" step); the YAML flag alone silently resets. Verify with an export round-trip; don't assume it stuck.
• Once enabled, internal scheduling tools are visible and deletable via the ADK CLI (known issue) — don't remove them by hand.
• Schedules themselves are created conversationally in chat, not in YAML.

3.3c Agent skills (NEW in 2.13.0) — live-verified 2.13.0

Agent skills are portable, version-controlled packages that equip agents with specialized knowledge and workflows. A skill is a SKILL.md file with YAML frontmatter, optionally accompanied by a sibling WXO.yaml (sent automatically on import) plus scripts/reference docs.

Skill vs. Knowledge base vs. Tool — which to reach for

These three attach to an agent but answer different questions. Picking wrong is the most common design mistake now that skills exist. Rule of thumb by what you're adding:

Decision shortcuts:

• "The agent keeps doing the steps in the wrong order / skipping a policy" → skill (encode the procedure), not a tool.
• "The agent needs to know facts from our docs" → knowledge base, not a skill (don't paste a document corpus into SKILL.md).
• "The agent must fetch or change something" → tool, always.
• They compose: a skill can describe when to call a tool or search a KB; it doesn't replace either. The Dr. House team uses all three — diagnostics-protocol (skill: the differential workflow + escalation rules), differential_diagnosis/lab_reference (tools: the actions), and a KB would be the place for, say, an ingested formulary.
• Skills vs. instructions: — put stable, reusable, shareable procedure in a skill (versioned, attachable to many agents); keep this agent's voice and one-off routing in instructions:.

---
name: diagnostics-protocol        # ⚠ kebab-case ONLY (agentskills.io spec) — see below
description: >-
  When and how to use this skill (used for relevance). Structured differential-diagnosis
  protocol for the diagnostics team.
version: 1.0.0
---

# Diagnostics Protocol
...the procedure the agent should follow...

Lifecycle (group is skills):

orchestrate skills import -f skills/diagnostics_protocol/SKILL.md --upsert   # or -d <dir> [-r]
orchestrate skills list                     # Name · Description · Mode · Tools · Scripts · ID
orchestrate skills get -n diagnostics-protocol
orchestrate skills upload-script    …       # attach an executable script to a skill
orchestrate skills upload-reference …       # attach a reference doc
orchestrate skills export -n diagnostics-protocol -o <dir|zip>
orchestrate skills remove -n diagnostics-protocol

Attach to a native agent by name via the new skills: field (round-trips on export):

skills:
  - diagnostics-protocol

⚠ Skills cost a turn at runtime (live-verified 2.13.0). Attaching a skill adds a real load_skill tool call to the run — it shows up in step_history as {"skill_name": "diagnostics-protocol", "params": {}} followed by a tool_response, and the skill body is then loaded into the system prompt. So a skill is not free: it costs a turn, input tokens on every subsequent generation, and a little latency. Attaching six skills to one agent is a real cost. Budget for it, and don't be surprised to see load_skill in traces.

⚠ Naming gotcha (live-verified): unlike every other wxO resource (which is snake_case), an agent-skill name must be kebab-case — lowercase letters, digits, and single hyphens only, no underscores, no leading/trailing/consecutive hyphens. Importing diagnostics_protocol fails with "Skill name … is invalid … (agentskills.io spec)"; use diagnostics-protocol. Import is upsert-by-name (same name updates in place; --upsert makes that explicit). agents list now shows Skills and Plugins columns.

3.4 Import (dependency-ordered)

orchestrate env activate local

# 1) connections first (tools/agents reference them)
orchestrate connections import -f connections/my_api.yaml

# 2) custom models (if any)
orchestrate models import -f models/granite.yaml --app-id watsonx_credentials

# 3) knowledge bases
orchestrate knowledge-bases import -f knowledge_base/kb.yaml

# 4) tools — link credentials with --app-id; python tools take -r
orchestrate tools import -k python -f tools/weather.py -r tools/requirements.txt
orchestrate tools import -k python -f tools/api_tool.py --app-id my_api

# 4b) MCP toolkits — group is `toolkits` (plural). `add` for inline config,
#     `import` for a pre-written MCP spec file. (see references/mcp-toolkits.md)
orchestrate toolkits add -k mcp -n my_toolkit --description "My MCP tools" \
  --package-root ./mcp_server --language node \
  --command '["node","dist/index.js","--transport","stdio"]' --tools "*"

# 5) agent skills (2.13.0) — before the agent that references them (§3.3c)
orchestrate skills import -f skills/diagnostics_protocol/SKILL.md --upsert

# 6) the agent
orchestrate agents import -f agents/weather_agent.yaml

# 7) controls LAST (2.15.0) — they bind to agents/tools/models by name, so those
#    must exist first. `controls import` is create-only: remove before re-running.
orchestrate controls remove -n weather_pii_guard 2>/dev/null || true
orchestrate controls import -f controls/weather_pii_guard.yaml

tools import -k accepts python | openapi | flow | langflow. Use --safe on tools/agents/knowledge-bases import to be prompted before overwriting an existing resource.

Never run orchestrate import directly. Put this code in import-all.sh instead and run that script.

3.5 Test

orchestrate agents list                                 # confirm it imported; see real `name`s
orchestrate agents list -v                              # full JSON incl. ids

Always parse agents list -v as JSON — never read the table. The rich table wraps every column to roughly one character per line at normal terminal width, so a tenant with more than a handful of agents produces hundreds of lines of unreadable output. The same applies to tools list, skills list and traces search. Where a command has -f/--format json use it; otherwise set COLUMNS=200 to widen the render.

• Reference agents/tools by their name (snake_case), never the display name. e.g. an agent shown as "FM - Aegis" may have name: FM_3009a0 — -n "FM - Aegis" will fail. Find the real name with agents list -v, or export it: orchestrate agents export -n <name> --kind native -o agent.yaml --agent-only.
• MCP-toolkit tools appear namespaced in listings as toolkit_name:tool_name.

Verify before handover (post-deploy gate)

Deployed ≠ verified. Never report an agent as "done" or "ready for handover" until it has been tested — or the human explicitly declined.

To test the agent, Orchestrate assets have to be imported/deployed first: Run import-all.sh.

After deploy, ask first, then prove it works:

"<agent> is deployed to <env>. Want me to smoke-test it before handover? I'll run 1 single-turn + 1 multi-turn test against <env> — this sends real prompts and may invoke its tools, so I'll keep to read-only prompts." — Yes / No

If yes, derive the tests from the agent's own spec (description, instructions, tools, and especially starter_prompts — those are example user prompts). Make the follow-up context-dependent (e.g. "which movie was it written for?" — "it" only resolvable from turn 1).

Important: Always run one single-turn test and additionally one multi-turn test:

1. Single-turn test
2. Multi-turn test 2a. first user input (this is input is different from 1. Single-turn test) 2b. second user input (and thread_id from 2a)

There are two ways to run these tests dependent on whether the watsonx-orchestrate-adk MCP server is available.

1. watsonx-orchestrate-adk MCP server is available

Run the MCP server watsonx-orchestrate-adk tool chat_with_agent for single-turn and multi-turn conversations.

2. watsonx-orchestrate-adk MCP server is not available

# Single-turn (1)
./.bob/skills/watsonx-orchestrate/references/wxo-chat.sh -n <agent> "<derived single-turn prompt>"
# → { "thread_id": "3f92692d-...", "final_message": "...", ... }

# Multi-turn (2a) — initial user input
./.bob/skills/watsonx-orchestrate/references/wxo-chat.sh -n <agent> "<derived multi-turn prompt>"
# → { "thread_id": "3f92692d-...", "final_message": "...", ... }

# Multi-turn (2b) — resume
./.bob/skills/watsonx-orchestrate/references/wxo-chat.sh -n <agent> --thread-id <thread-id> -r "<second user input>"
# → { "thread_id": "3f92692d-...", "final_message": "...", "reasoning_trace": {"steps": [...]}, ... }

Pass = behavior, not exact text (LLMs are non-deterministic): no error, on-topic, the expected tool was called (visible via -r; -l for custom agents), and the multi-turn follow-up uses prior context. Then emit a short temp/TEST_REPORT.md (prompts, response excerpts, tool-call evidence, env, timestamp, pass/fail) and report status honestly: "deployed and tested (2/2)", "deployed; test 2 failed — …", or "deployed; not tested at your request".

Safety: keep tests read-only by default — if the agent has READ_WRITE tools, do not craft prompts that trigger writes unless the human opts in. Full recipe + report template: references/testing-debugging.md.

3.6 Debug → Publish

Debugging in §6; promoting to a production instance in §7. Run the §3.6 gate (or record that the human declined) before §7 handover.

4. Models / LLMs

• List what the active environment offers: orchestrate models list.
• Reference models in agent YAML by their full id, e.g. watsonx/meta-llama/llama-3-3-70b-instruct, watsonx/ibm/granite-3-3-8b-instruct, or a gateway provider like groq/openai/gpt-oss-120b.
• Use groq/openai/gpt-oss-120b as default model if available.
• The experimental_customer_care agent style expects groq/openai/gpt-oss-120b.
• Premier models are disabled by default — enable them at the tenant level before referencing one, or models list won't surface it (the list legend marks premier with $). 2.13.0 adds GPT-5.4 and other premier models. Commands (live-verified 2.13.0):

  orchestrate models config are-premier-models-enabled   # → "Premier models enabled: False"
  orchestrate models config enable-premier-models        # tenant-wide opt-in
  orchestrate models config disable-premier-models

• New provider redhat-ai (2.14.0). The release note calls it red_hat_ai; the actual ModelProvider enum value is redhat-ai — use that in kind: model YAML. Full provider list in references/connections-models-kb.md.
• External agents gained the msftstudio provider (Microsoft Copilot Studio; the release note calls it microsoft_copilot_studio) alongside salesforce, external_chat, wx.ai and A2A 0.2.1/0.3.0.
• Prebuilt domain agents are multi-provider: gpt-oss-120b via Groq on AWS/IBM Cloud, via watsonx.ai on on-prem/GovCloud — pick the id that models list actually shows for your deployment.
• To add your own watsonx.ai model you create a watsonx_credentials connection, then a kind: model YAML, then orchestrate models import. Full walkthrough (provider schema, space/project/deployment, CPD on-prem) is in references/connections-models-kb.md.

4a. Controls — policy guardrails on agents, tools and models (NEW in 2.15.0)

A control binds a policy artifact (a PII filter, a content guardrail, a rate limiter…) to one or more agents, tools or models, at a named execution hook. This is how you put a hard, platform-enforced boundary around an agent instead of asking the model nicely in instructions:.

orchestrate controls list-types                    # what policy artifacts exist here
orchestrate controls get-type -n pii_filter -v     # → the artifact's JSON config_schema
orchestrate controls create -a "PII Filter" -n house_pii_guard \
  --hook agent_pre_invoke --hook agent_post_invoke --priority 50 \
  --config '{"detect_ssn": true, "detect_email": true, "default_mask_strategy": "redact"}' \
  --agent dr_house
orchestrate controls list [--agent X|--tool Y|--model Z|--artifact A] [-v]
orchestrate controls get-details -n house_pii_guard -v
orchestrate controls count                         # totals per asset type
orchestrate controls update  -n house_pii_guard --priority 10 --config '{…}'
orchestrate controls export  -n house_pii_guard -o control.yaml
orchestrate controls import  -f control.yaml
orchestrate controls remove  -n house_pii_guard

The release note lists list, list-types, get-type, create, delete. The real CLI has ten commands and the delete verb is remove, not delete.

Execution hooks (server-validated — an invalid one is a clean 422 naming all six): agent_pre_invoke, agent_post_invoke, tool_pre_invoke, tool_post_invoke, prompt_pre_fetch, prompt_post_fetch. --priority orders them; lower runs first (default 100).

Policy artifacts on IBM Cloud SaaS (list-types, live 2.15.0). Each is valid only for the asset types listed — get-type -n <name> -v prints its asset_type and full config_schema:

It genuinely enforces — A/B proof (live-verified 2.15.0)

Same agent, same prompt, control on vs. off:

The filter rewrites the payload at agent_pre_invoke, so the model never sees the PII and declines on its own. Two consequences worth knowing: the refusal looks like a model refusal, not a policy block (check the trace before you "fix" the prompt), and the redaction is visible in observability traces export, which is the fastest way to confirm a control is live.

Four things controls accepts silently — all live-verified

These fail quietly, at create time, and only show up as a control that does nothing:

1. --config is NOT validated against config_schema. A bogus key (mask_strategy instead of default_mask_strategy) and a bogus enum value ("NOT_AN_ENUM_VALUE") were both stored verbatim, HTTP 200. Read get-type -n <artifact> -v and copy the key names exactly — a typo means the control runs with schema defaults, i.e. everything off. (Contrast: --hooks is validated.)
2. Asset-type mismatches are accepted. RateLimiterPlugin is asset_type: ["tool"], yet binding it to an --agent on an agent hook succeeded. Check asset_type yourself.
3. A control with no binding at all is accepted — it counts in controls count totals but appears under no asset bucket. If Total exceeds Agent + Tool + Model, you have orphans.
4. controls import is create-only, not upsert. Re-running it fails with 400 Policy binding with name "X" already exists — there is no --upsert. In import-all.sh, controls remove -n X first (ignoring failure) or use controls update.

⚠ export → import does not round-trip

controls export writes the agent's display name:

agent_names:
- Dr. House (Diagnostics Lead)      # ← what export emits

…but controls import resolves agent_names against the snake_case name, so re-importing your own export fails:

No agent found with name 'Dr. House (Diagnostics Lead)'

Fix: rewrite agent_names to snake_case before importing (- dr_house). With that one substitution everything else round-trips faithfully — hooks, priority, config, and artifact_name given as either the display name (PII Filter) or the internal name (pii_filter). Keep hand-written control YAML in Git; treat export as a read-only inspection tool.

Authoring format (what import actually wants):

spec_version: v1
kind: control
control:
  artifact_name: pii_filter          # internal name or display name
  name: house_pii_guard
  display_name: House PII Guard
  description: Redact patient PHI on the diagnostics agent
  hooks: [agent_pre_invoke, agent_post_invoke]
  priority: 50
  config:
    detect_ssn: true
    default_mask_strategy: redact
    redaction_text: "[PHI-REDACTED]"
  agent_names: [dr_house]            # snake_case! also tool_names / model_names

controls update replaces rather than merges: --hooks, --config, --agent, --tool and --model each overwrite the previous value wholesale, while any flag you omit is preserved. Each update bumps the control's version.

4b. Voice configuration (2.14 → 2.15 additions)

orchestrate voice-configs import|list|get|export|remove. Attach to an agent with voice_configuration: <name>.

⚠ Provider strings are <vendor>_stt / <vendor>_tts, not the bare vendor. provider: deepgram fails server-side with the misleading deepgram_config must be specified for deepgram — the correct value is deepgram_stt. The ADK models provider as a free string with no enum, so a wrong value only surfaces as a 422 from the platform.

spec_version: v1
kind: voice_configuration
name: front_desk_voice
speech_to_text:
  provider: deepgram_stt
  deepgram_stt_config:
    api_key: …
    api_url: wss://api.deepgram.com/v2/listen   # REQUIRED — omitting it is a pydantic error
    model: flux-general-en          # 2.15.0 fix: flux-* no longer raises a false warning
    language: en
text_to_speech:
  provider: deepgram_tts
  deepgram_tts_config:
    model: aura-2-zeus-en
    normalize_volume: true          # 2.14.0 — auto-levels the output audio
user_idle_handler:
  enabled: true
  idle_timeout: 8                   # seconds before the handler fires
  idle_max_reprompts: 2
  idle_timeout_message: Are you still there?
  idle_hangup_message: I will close the line for now.
  use_llm_generated_idle_message: false   # 2.15.0 — false = use the static message above
  repeat_previous_message: false          # 2.15.0 — false = don't replay the last agent turn
agent_idle_handler:
  typing_enabled: true
  typing_duration_seconds: 5
  audio_clip_id: guitar_1           # guitar_1 | listen_1 | silence | custom
  hold_audio_seconds: 15
  long_running_task_seconds: 3      # 2.15.0 — processing time before the pre-hold message
  pre_hold_message: Pulling the chart now, one moment.
  hold_message: Still working on it. Thanks for your patience.

All of the above round-trips through voice-configs export (API keys are stripped).

⚠ Google TTS (2.14.0) was rejected by the SaaS backend at time of writing. The ADK serialises google_tts_config correctly (verified against TextToSpeechConfig), but the platform answered 422 google_tts_config must be specified for google_tts. Treat Google TTS as doc-sourced, not live-verified, and confirm on the target tenant before promising it. Its fields: voice, language, speaking_rate (0.25–4.0), pitch (-20–20), volume_gain_db (-96–16), ssml_gender, effects_profile_id, plus exactly one of credentials_json / api_key.

5. Critical constraints (these cause silent failures — internalize them)

Python tools

• ✅ Every callable that becomes a tool must be decorated with @tool.
• ✅ Use Google-style docstrings: a summary, then Args: (each as name (Type): desc), then Returns: (Type: desc). No blank line between the Args: and Returns: blocks — extra blanks break the parser.
• ✅ Every parameter and the return value must have type hints, matching the docstring.
• ✅ Each tool file must be self-contained: only stdlib, common third-party (requests, pydantic, …), and ibm_watsonx_orchestrate imports. No cross-file local imports (from .utils import x / from tools.shared import y).
• ✅ Credentials are never function parameters. Declare expected_credentials=[…] on @tool and fetch at runtime via ibm_watsonx_orchestrate.run.connections.
• ✅ Define Pydantic models as explicit classes — never type(...) dynamic creation.
• ✅ Never add ibm-watsonx-orchestrate itself to a tool's requirements.txt — the runtime provides it.

Flows

• ✅ Signature is exactly def build_<flow_name>(aflow: Flow) -> Flow: — param named aflow, returns the flow, function name starts with build_.
• ✅ One flow per file. Wire with aflow.sequence(START, …, END) / aflow.edge(...).
• ✅ map_output / map_input expressions are single-line — no function defs/calls.
• ✅ aflow.prompt(...) requires a system_prompt.

Agent YAML

• ✅ spec_version: v1 and kind: native are mandatory — omitting them fails import.
• ✅ tools/collaborators/knowledge_base/skills list resources by name; the referenced resources must already be imported.
• ✅ style: default is react_core (2.13.0). default, react, planner still import but log a deprecation warning — prefer react_core.
• ✅ skills: names are kebab-case (agentskills.io spec) — every other resource name is snake_case, but a skill named with an underscore is rejected (§3.3c).
• ✅ toolkits are only valid for experimental_customer_care style (and the schedulable-agent exception).

Connections YAML

• ✅ kind: connection (singular). Inside environments.<env> use security_scheme: (not kind:). At minimum a draft environment.
• ✅ OAuth2 needs auth_type: oauth2_auth_code (not authorization_code).
• ✅ YAML defines structure; real secrets are set with orchestrate connections set-credentials. Never hardcode secrets.
• ⚠ Custom API-key header (2.15.0) — accepted by the CLI, BROKEN on the SaaS backend. connections configure -a <app> --env draft -t team -k api_key -u <url> --name "X-Auth-Token" is supposed to replace the default api_key header. Live-verified 2.15.0 on IBM Cloud SaaS: the create path reports success but stores nothing (connections export emits no name), and the update path fails outright:

  500 CM-UNKNOWN-001 … details: column "name" of relation
      "application_connection_configs" does not exist
  

  The same update without --name succeeds, so the flag itself is the trigger — the backend schema has not caught up with the CLI. Do not depend on a custom header name until you have verified it on the target tenant. If the API needs a non-default header, fall back to --kind key_value with an explicit entry, or send the header from the tool code. (--name is also rejected outright on any kind other than api_key.)

Controls (2.15.0) — full section in §4a

• ✅ --config is not validated: copy key names from controls get-type -n <a> -v.
• ✅ --hooks is validated; --priority is lower-runs-first.
• ✅ controls import is create-only — remove first when re-running an import script.
• ✅ controls export emits agent display names; rewrite them to snake_case before import.

Remember: After code generation always deploy/import and runs tests (see 3.5 Test)

6. Debugging playbook

6a. Observability & cost — what wxO gives you, and what it doesn't

Every completed run carries a trace_id. That is your handle on everything below. Full field maps, the span-tree rule and the cost model: references/agentops-evaluations.md §4–§7.

Three surfaces, increasing detail — pick deliberately:

Native (just read it): end-to-end and per-span latency · per-call token counts · model id · sessionId/userId · the span tree · sampled LLM-judge scores[] · routing accuracy from evaluations.

Not native (you build it):

• Cost. trace.totalCost returns 0 — no price sheet is attached. Tokens are exact; pricing is yours. Multiply usage by a rate card keyed on observation.model. Expect input tokens to dominate (~9:1 measured on a multi-agent session) — delegation re-sends context, and loaded skills sit in every prompt.
• Error rate, tail latency, tool-call success. Derive from runs and traces. Exclude the sub-second infrastructure traces (blank agent name) that traces search returns.
• Dashboards. None ship. No error-rate panel, no latency histogram, no tool-success view. State this in scoping before a customer assumes otherwise.

Latency triage in one rule: sort observations by latency, rebuild the tree with parentObservationId, then — outer span ≫ inner span → the time is in the platform (cold start, queueing); outer ≈ inner → the time is in your code or what it calls. The two have completely different fixes, and only the v3 API can tell them apart.

⚠ Fields on the v3 API are camelCase (startTime, totalCost, parentObservationId) — unlike every other wxO payload. Snake_case guesses silently return None.

Iterate fast: edit → re-import (overwrites by name) → re-test in chat. Use orchestrate agents export -n <name> --kind native -o agent.yaml --agent-only to snapshot a working definition into Git, and --safe on imports to avoid clobbering.

More: references/testing-debugging.md.

6b. Observability for agents running OUTSIDE wxO (2.14 → 2.15)

An agent that runs on your own infrastructure (LangGraph, CrewAI, a plain script) can now push its execution traces into the wxO Agent Analytics dashboard, so external and native agents show up side by side. Two routes:

1. OpenTelemetry export — any language, any framework, standard OTel SDK:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Point the OTLP/HTTP exporter at <instance-url>/v1/orchestrate/inject/traces with Authorization: Bearer <token> (exchange your API key at the IAM/MCSP token endpoint). Environment it expects: TENANT_ID (<account-id>_<instance-id>), AGENT_ID, ENVIRONMENT_NAME (draft/live), and OTEL_RESOURCE_ATTRIBUTES="tenant.id=$TENANT_ID,deployment.environment=$ENVIRONMENT_NAME". Set agent.id, langfuse.session.id and langfuse.user.id on the root span — those are what the dashboard groups on. Always force_flush() before the process exits or the last trace is lost.

2. Observability SDK — decorator-based, aimed at LangGraph:

pip install -i https://test.pypi.org/simple/ ibm-watsonx-orchestrate-sdk   # Test PyPI pre-release

from ibm_watsonx_orchestrate_sdk.client import Client
from ibm_watsonx_orchestrate_sdk.observability import Tracer, TracerConfig, register_tracer
from ibm_watsonx_orchestrate_sdk.observability.decorators import (
    configure_tracing, trace_agent_call, trace_llm_call, trace_tool_call, trace_call,
)

TracerConfig(client=…, agent_id=…, workspace_id=…, environment="live", tenant_id=…), then register_tracer(tracer) once at module load. Decorate the graph factory with @configure_tracing, the top node with @trace_agent_call, and LLM/tool calls with @trace_llm_call / @trace_tool_call. Automatic token refresh, root-span creation, tool attributes and PII decorators are built in; export also works to any OTEL backend (Jaeger, Instana).

⚠ Both are doc-sourced, not live-verified here. The SDK ships from Test PyPI as a pre-release — pin it and re-check the package name before putting it in a customer build.

7. Publishing to production

There is no publish verb — publishing = activating the target environment (SaaS or on-prem, see §2a), re-importing the artifacts there, and deploying.

Never hand over an untested agent. Run the §3.6 verification gate after deploy (or record that the human declined). "Deployed" is a fact about the platform; "verified" is a fact about behavior — only the latter is handover-ready.

# 1) Activate the target env (registered once per §2a)
orchestrate env activate prod         # SaaS or on-prem

# 2) Re-import the same artifacts (connections → models → KB → tools → agent),
#    setting that env's credentials: orchestrate connections set-credentials ...
./import-all.sh

# 3) Deploy / retire an agent
orchestrate agents deploy   -n weather_agent
orchestrate agents undeploy -n weather_agent

Keep one set of YAML/Python artifacts; only connection credentials and model provider_config differ per environment. Version the artifacts in Git and treat import-all.sh as the source of truth for promotion across dev → SaaS/on-prem. orchestrate channels exposes a deployed agent on channels such as embedded web chat.

Embedded web chat (the channel for chat_with_docs uploads)

Generate a drop-in <script> snippet for a deployed agent:

orchestrate channels webchat embed --agent-name <agent> --env live   # or draft

Paste the snippet into a page with a <div id="root">; it loads <hostURL>/wxochat/wxoLoader.js?embed=true and renders the launcher. This is the right surface for chat_with_docs — the upload widget here triggers ingestion, which the raw runs API does not (§8).

⚠ CRN gotcha (live-verified 2.12.0 SaaS): the command auto-fetches the instance CRN from the IBM Cloud resource-controller, which 403s with an instance-scoped API key and then prompts interactively (aborts in scripts). The CRN is in the bearer token's unique_instance_crns claim — extract it and pipe it in:

CRN=$(python -c "import yaml,os,json,base64;t=yaml.safe_load(open(os.path.expanduser('~/.cache/orchestrate/credentials.yaml')))['auth']['<env>']['wxo_mcsp_token'];p=t.split('.')[1];p+='='*(-len(p)%4);print(json.loads(base64.urlsafe_b64decode(p))['unique_instance_crns'][0])")
echo "$CRN" | COLUMNS=400 orchestrate channels webchat embed --agent-name <agent> --env live

Paste the emitted snippet into any page with a <div id="root"> to render the launcher.

⚠ Two more webchat gotchas (live-verified 2.13.0):

• COLUMNS=400 is not optional. The CLI renders the snippet through rich at your terminal width, so without it the JavaScript arrives hard-wrapped mid-string (the CRN and orchestrationID split across lines) and the page fails silently. Always widen, then extract with sed -n '/<script>/,/<\/script>/p'.
• The embed 401s from an unauthenticated origin. The loader fetches, wxoLoader.init() runs and the launcher renders — then every backend call returns 401 and the chat never connects:

  [warn]  [WxOChat] Remote landing not available in this environment
  [error] Failed to load resource: the server responded with a status of 401 ()
  

  This is not a missing channel: channels create --type webchat explicitly refuses with "Webchat channels cannot be created using the 'create' command. Webchat is automatically available for all agents.", and channels list-channels legitimately reports "No channels found". It is authentication — the embedded chat expects an authenticated session against the tenant. Serving it to anonymous users is configured in the watsonx Orchestrate UI (web chat security / allowed origins); the ADK does not expose it. Budget for this: "paste the snippet and it works" holds only inside an authenticated context.

8. Embedding agents in your application (runtime REST API)

This is for consuming a deployed agent from your own app — a web/mobile backend, service, IDE, or another agent owns the UX and calls wxO like any backend service. It is distinct from the embedded web-chat widget (that's a channel, §7 / orchestrate channels, a drop-in UI you don't code against).

wxO exposes a runtime REST API under <service-url>/api/v1, bearer-token auth:

• /orchestrate/{agent_id}/chat/completions — OpenAI-compatible; drop-in for anything that already speaks OpenAI Chat Completions. Reply at choices[0].message.content.
• /orchestrate/runs (+ GET /orchestrate/runs/{run_id}) and /orchestrate/runs/stream — richer/async: tool-step outputs, decoding params (greedy / temperature: 0), guardrails, usage. Reply at result.data.message.content[0].text.
• /completions, /completions/chat — raw Gateway model, no agent/tools.

Both agent families return a thread_id — send it back to continue a conversation (multi-turn memory). The Python SDK RunClient

Truncated - read the full file at https://github.com/thomassuedbroecker/galaxium-travels-watsonx-orchestrate-mcp-bob-guide/blob/8d6e5dd39236bb57b78e24a22378890d4736cf04/.bob/skills/watsonx-orchestrate/SKILL.md.