eval-monitor

Imported from reza-microsoft/reza-aifsdk (agento/.claude/skills/eval-monitor/SKILL.md). Install upstream with npx skills add reza-microsoft/reza-aifsdk --skill eval-monitor. Copyright stays with the author.

Given a valid webeval_next/scripts/run_evals.sh command in $ARGUMENTS, take it all the way from "launch" to "AML dashboard has the histograms," detecting and killing the runs that hang at 100% completion, and never skipping post-eval analysis — even on kill.

This skill exists because:

1. A webeval run commonly hangs near 100% completion (last few trajectories wedged in retry loops, playwright browsers that never finalize, vLLM that stops responding). Sitting indefinitely is a real cost; the partial results are basically done and worth analyzing.
2. When the eval CLI is killed, post_eval_analyzers never fire — so post_eval_report.txt is missing and the AML error-histogram metrics never get logged. Every subsequent ingestion and every dashboard comparison assumes those metrics are present. They're not optional.

Environment — non-negotiable

All Python in this skill runs under the webeval_next env. Activate whichever form your box has set up — the webeval_next conda env, or the webeval_next/.venv uv venv.

Symptom of the wrong env: ModuleNotFoundError: No module named 'aztool' from aztool.az_vllm during --start-vllm.

All commands run from <repo_root>/webeval_next/ (resolve at start — there may be multiple worktrees).

Phase 0 — parse arguments

$ARGUMENTS contains (in order):

1. Optional leading flag: --retry-blocked-on-complete — when present, after the main run finishes (either naturally or by stuck-kill + post-eval), relaunch the same command with --browserbase and --max-retries <2× original> to give blocked sites (united.com, amazon.com anti-bot, ign.com, …) a second chance through a clean residential IP. The second run reuses the same run_id so webeval skips the already-completed tasks and only re-runs the failed ones.
2. The full ./scripts/run_evals.sh ... command, verbatim.

From the command, extract (required):

• -r <RUN_ID> — used to locate log file and output dir
• -b <BENCHMARK> — single benchmark only; the monitor design below is per-benchmark. If the user passed multiple benchmarks, stop and ask which one to monitor.
• -s <SYSTEM>
• --start-vllm presence — tells us whether to watch vLLM startup before eval log appears
• --max-retries <N> — remember the original for the --retry-blocked-on-complete re-run
• --model-url <path_or_uri> — informational
• --device-id <ids> — informational

Compute:

• EVAL_LOG=./logs/runid_${RUN_ID}_${BENCHMARK}.log
• LAUNCH_LOG=./logs/launch_${RUN_ID}_${BENCHMARK}.out

Pre-flight: refuse to run if RUN_ID is empty — default -r default would overwrite a prior run's output.

Phase 1 — attach-or-launch

Always check for an existing run first — relaunching over one that's already in flight causes port collisions on --vllm-port / --model-port and wastes the work already done. The -r RUN_ID + -b BENCHMARK tuple uniquely identifies a run on this host.

cd "$(git rev-parse --show-toplevel)/webeval_next"
# activate the webeval_next env (conda `webeval_next` or `webeval_next/.venv`)
mkdir -p logs

# 1) Detect existing run_evals.sh matching both RUN_ID and BENCHMARK.
#    Both must match to avoid attaching to a different benchmark that happens
#    to share the run_id (rare but possible).
#    NOTE: do NOT `| head -1` here — if >1 match we must stop and ask, not
#    silently pick the first (see "Attach edge cases" below).
MATCHES=$(pgrep -af "run_evals.sh" | awk -v r="$RUN_ID" -v b="$BENCHMARK" '
    $0 ~ ("-r[= ]+" r "( |$)") && $0 ~ ("-b[= ]+" b "( |$)") { print $1 }')
N_MATCHES=$(printf '%s\n' "$MATCHES" | grep -c .)

if [ "$N_MATCHES" -gt 1 ]; then
    echo "[attach] AMBIGUOUS: $N_MATCHES run_evals.sh processes match run_id=$RUN_ID benchmark=$BENCHMARK:"
    pgrep -af "run_evals.sh" | awk -v r="$RUN_ID" -v b="$BENCHMARK" '
        $0 ~ ("-r[= ]+" r "( |$)") && $0 ~ ("-b[= ]+" b "( |$)")'
    echo "Stop here and ask the user which PID to attach to before proceeding."
    exit 1
fi

EXISTING=$MATCHES

if [ -n "$EXISTING" ]; then
    echo "[attach] existing run_evals.sh pid=$EXISTING for run_id=$RUN_ID benchmark=$BENCHMARK"
    # Sanity: confirm log file exists (proves it's past CLI arg parsing).
    # If $EVAL_LOG is missing but the process is alive, the run is still in vLLM startup;
    # that's fine — Phase 2 will see [vllm-ready] when it emits.
    ls -la "$EVAL_LOG" "$LAUNCH_LOG" 2>/dev/null || echo "  (logs not yet populated — run is in early phase)"
    # Skip the nohup launch; jump straight to Phase 2 monitoring.
else
    nohup <full run_evals.sh command from $ARGUMENTS> > "$LAUNCH_LOG" 2>&1 &
    disown
    echo "LAUNCHER_PID=$!"
fi

Always echo the log paths to the user immediately after launch/attach, so they can tail along in another terminal. This is explicit user preference — they want to follow progress themselves, not just wait for the skill's periodic reports.

cat <<EOF

=== eval-monitor: tail these to follow progress ===
  EVAL_LOG   : tail -F $EVAL_LOG
  LAUNCH_LOG : tail -F $LAUNCH_LOG
  VLLM_LOG   : tail -F \$(ls -t logs/vllm_server_${RUN_ID}_*.log 2>/dev/null | head -1)
EOF

Attach edge cases to handle:

• $LAUNCH_LOG was written by the prior launch under a different path (e.g., user launched manually with stdout redirected to /tmp/eval_<runid>.log). The Phase 2 monitor's [launch-error] / [vllm-ready] greps will silently never fire. Before starting Phase 2, confirm $LAUNCH_LOG is non-empty and contains recognizable run_evals.sh banner output (e.g., Configuration:, Running benchmark:). If it's empty, the user's launch wrote elsewhere — ask them for the actual log path and point $LAUNCH_LOG at it. Do NOT invent one.
• No $EVAL_LOG yet but attached: benign — the eval log is created by webeval.cli after vLLM is up. Monitor polls every 180s and will pick it up.
• Found more than one match: stop and ask the user which PID to attach to. Ambiguous attach has silently targeted the wrong run before.

Confirm the launcher and run_evals.sh child both appear (fresh launch only):

sleep 5 && pgrep -af "run_evals.sh.*${RUN_ID}" | head -5

Verify cwd of the run_evals.sh process points to webeval_next/:

ls -l /proc/<pid>/cwd

Bash tool calls don't share cwd between invocations, so this check matters.

Phase 2 — monitor with the Monitor tool

Arm a persistent Monitor with a polling script that emits events only on state changes (not a per-tick heartbeat — the user doesn't want spam). It must cover every terminal state the skill will act on: vLLM ready, progress delta, stall near completion, crash, done.

The evaluated-count regex ([Evaluation .+] Completed: score=) is the ground truth for progress. Started-count ([Execution .+] Start) — deduped by task id — is not a reliable denominator: on a resumed run (same -r RUN_ID as a prior attempt) cached/already-scored tasks never fire [Execution Start], so started_unique plateaus at (tasks this run actually re-executed) — typically far below the benchmark's true task total. So the stuck-near-complete signal uses an absolute count floor near the benchmark total (NEAR_COMPLETE_COUNT), not a percentage of started_unique. Benchmark totals (verified from completed runs):

• webvoyager: ~595 tasks → NEAR_COMPLETE_COUNT=580
• om2w (OnlineM2W_03192026): 300 tasks → NEAR_COMPLETE_COUNT=270
• other benchmarks: measure from a prior completed run's is_aborted == True/False totals in post_eval_report.txt, or ask the user

Stale thresholds also tolerate webvoyager/om2w's long browser-retry tail: 1 hour (STALE_NEAR=3600) of no progress at near-complete before [stuck-near-complete], and 1.5 hours (STALE_GENERAL=5400) for a plain [stuck] signal elsewhere. Earlier 15-min defaults caused false positives on the tail.

REPO=$(git rev-parse --show-toplevel)
LOG=$REPO/webeval_next/logs/runid_<RUN_ID>_<BENCHMARK>.log
LAUNCH=$REPO/webeval_next/logs/launch_<RUN_ID>_<BENCHMARK>.out
# Per-benchmark: set to a count just below the benchmark's true total.
# See list above for known benchmarks.
NEAR_COMPLETE_COUNT=<fill-in-per-benchmark>
STALE_NEAR=3600       # 1 hour stale at near-complete before recommending kill
STALE_GENERAL=5400    # 1.5 hours stale elsewhere before emitting plain [stuck]
last_count=-1
last_started=-1
last_change=$(date +%s)
vllm_ready=0
done_reported=0
started=$(date +%s)
err_seen=0
mlflow_broken=0

while true; do
  now=$(date +%s); elapsed=$((now - started))

  # vLLM readiness (once)
  if [ $vllm_ready -eq 0 ] && grep -qE "VLLM server is ready" "$LAUNCH" 2>/dev/null; then
    echo "[vllm-ready] after ${elapsed}s"
    vllm_ready=1; last_change=$now
  fi

  # Launch-level fatal errors (once)
  if grep -qiE "ModuleNotFoundError|No module named|did not start within" "$LAUNCH" 2>/dev/null && [ $err_seen -eq 0 ]; then
    echo "[launch-error] $(grep -iE 'ModuleNotFoundError|No module named|did not start within' "$LAUNCH" | tail -1)"
    err_seen=1
  fi

  if [ -f "$LOG" ]; then
    # evaluated = numerator; started-unique = denominator approximation
    count=$(grep -cE "\[Evaluation .+\] Completed: score=" "$LOG" 2>/dev/null || echo 0)
    started_n=$(grep -oE "\[Execution [^]]+\] Start" "$LOG" 2>/dev/null | sort -u | wc -l)
    err_n=$(grep -cE "CUDA out of memory|Killed$|OutOfMemory" "$LOG" 2>/dev/null || echo 0)

    if [ "$count" != "$last_count" ] || [ "$started_n" != "$last_started" ]; then
      echo "[progress] evaluated=$count started_unique=$started_n elapsed=${elapsed}s"
      last_count=$count; last_started=$started_n; last_change=$now
    fi

    # Stall detection — absolute count floor (not %-of-started; started_unique
    # undercounts on a resumed run, so a percentage would false-positive).
    stale=$((now - last_change))
    if [ $done_reported -eq 0 ]; then
      if [ "$count" -ge "$NEAR_COMPLETE_COUNT" ] && [ $stale -gt $STALE_NEAR ]; then
        echo "[stuck-near-complete] evaluated=$count (>= $NEAR_COMPLETE_COUNT); no new evaluation in ${stale}s (> ${STALE_NEAR}s) — recommend kill"
        done_reported=1  # prevent re-emitting; skill decides next action
      elif [ $stale -gt $STALE_GENERAL ]; then
        echo "[stuck] evaluated=$count; no progress in ${stale}s (> ${STALE_GENERAL}s)"
        last_change=$now  # cool down
      fi
    fi

    # Hard errors that warrant immediate surfacing
    if [ "$err_n" -gt 0 ]; then
      last_err=$(grep -E "CUDA out of memory|OutOfMemory|Killed$" "$LOG" | tail -1)
      echo "[fatal] $last_err"
    fi

    # MLflow / AML metric-push failure — SILENT in the code
    # (MlFlowRateLimiter.log_metrics catches the exception, prints once, and
    # continues). Without this detector, a 9-hour eval can finish "cleanly"
    # with exit code 0, post_eval_report.txt on disk, and ZERO histograms on
    # the AML dashboard because the server-side MLflow run expired mid-run.
    # See "silent MLflow upload failure" in the failure-modes table.
    if [ $mlflow_broken -eq 0 ]; then
      if grep -qE "Failed to log metrics|RESOURCE_DOES_NOT_EXIST|MlflowException|Run .* was not found" "$LAUNCH" "$LOG" 2>/dev/null; then
        n_fail=$(grep -cE "Failed to log metrics|RESOURCE_DOES_NOT_EXIST" "$LAUNCH" "$LOG" 2>/dev/null | awk -F: '{s+=$2} END {print s+0}')
        echo "[mlflow-push-failing] $n_fail metric-push rejections detected — AML dashboard will NOT receive histograms even on clean finish"
        mlflow_broken=1
      fi
    fi
  fi

  # Launcher died?
  if ! pgrep -f "run_evals.sh.*<RUN_ID>" > /dev/null 2>&1; then
    if [ $done_reported -eq 0 ]; then
      echo "[launcher-exited] after ${elapsed}s (evaluated=$last_count, started=$last_started)"
      done_reported=1
    fi
  fi

  sleep 180
done

Monitor config: persistent: true, timeout_ms: 3600000 (max). For runs > 1h, re-arm on expiry.

Filter coverage (critical). Ensure every terminal branch below has a matching emit: [vllm-ready], [progress], [stuck], [stuck-near-complete], [launcher-exited], [fatal], [launch-error], [mlflow-push-failing]. A silent monitor is indistinguishable from a running one.

Phase 3 — react to Monitor events

Be gentle on stuck-general. A single [stuck] after STALE_GENERAL (1.5h default) of no progress far from completion is more likely "a few long-tail tasks in their own browser retries" than a real hang. Only escalate after a second consecutive stall reading or obvious sign of process death. Near completion ([stuck-near-complete], count ≥ NEAR_COMPLETE_COUNT + 1h stale) skips this — that's almost always a hang.

Phase 4 — kill path (only when stuck or fatal)

Critical ordering: Kill the launcher + workers first, leaving vLLM alive. Phase 5 needs vLLM running so the eval_only re-invocation reaches the same system.hash() / model_host.hash() path and reads the correct traj/. Killing vLLM before Phase 5 corrupts the hash, points the CLI at an empty output folder, and uploads an all-zeros histogram to AML.

Proc-level identification uses explicit PID sets rather than pkill -f patterns that match $RUN_ID — the pattern can catch your own bash shell if it contains the run id on its command line, and that self-kills mid-kill.

# activate the webeval_next env (conda `webeval_next` or `webeval_next/.venv`)
cd "$(git rev-parse --show-toplevel)/webeval_next"

# 1) Capture PIDs once. Exclude our shell (contains $RUN_ID via args).
SHELL_PID=$$
WORKER_PIDS=$(pgrep -f "run_evals.sh.*${RUN_ID}\|python.*webeval.*${RUN_ID}\|python -m aztool.az_vllm.*${RUN_ID}" | grep -v "^${SHELL_PID}\$")
# Match vLLM by its *model path*, not the run id — the path is unique to this run.
VLLM_PIDS=$(pgrep -f "vllm.*$(echo "$MODEL_URL_BASENAME" | sed 's/[]\\/$*.^|[]/\\&/g')")

# 2) Launcher + CLI workers + az_vllm supervisor, leaving the vLLM server alive.
for p in $WORKER_PIDS; do kill -TERM "$p" 2>/dev/null; done
sleep 5
for p in $WORKER_PIDS; do kill -KILL "$p" 2>/dev/null; done

# 3) DO NOT kill vLLM yet — Phase 5 needs it. Jump to Phase 5 now.
#    After Phase 5 completes, come back here and finish teardown:
#       for p in $VLLM_PIDS; do kill -TERM "$p" 2>/dev/null; done
#       sleep 5; for p in $VLLM_PIDS; do kill -KILL "$p" 2>/dev/null; done

# 4) Orphaned browsers from the killed workers — safe to clean now.
#    Scope to ppid=1 (orphans) so we don't touch other runs' chrome children.
for p in $(ps -eo pid,ppid,comm | awk '$2==1 && $3 ~ /chrome/ {print $1}'); do
    kill -KILL "$p" 2>/dev/null
done

Do NOT remove the run's output directory. Post-eval in Phase 5 needs the traj/ folder.

Phase 5 — post-eval analysis (ALWAYS)

Goal: re-aggregate the error histograms and success metrics from verifier outputs already on disk (each trajectory's scores/<eval_hash>.json) and push them to the same AML dashboard — without re-invoking any verifier LLM.

Resolve the output folder FIRST — don't guess the path

The runs/<system_hash>/<model_hash>/<user>/<benchmark>/<run_id>/ path lives under a Hydra-generated /tmp/tmpXXXX/ root that is not predictable from the config. Guessing it (or re-deriving it via a fresh CLI invocation that might compute a different model_host.hash()) is how you end up analyzing an empty directory. The single reliable source of truth is the CLI's own log line:

[ErrorAnalyzer] wrote post-eval report to <PATH>/post_eval_report.txt

Grep it out of $LAUNCH_LOG before doing anything else in Phase 5:

OUTPUT_DIR=$(grep -oE '\[ErrorAnalyzer\] wrote post-eval report to [^ ]+/post_eval_report\.txt' "$LAUNCH_LOG" \
    | tail -1 | awk '{print $NF}' | sed 's|/post_eval_report\.txt$||')
[ -d "$OUTPUT_DIR/traj" ] || { echo "FATAL: $OUTPUT_DIR/traj not found"; exit 1; }
echo "output_folder = $OUTPUT_DIR"
echo "traj count    = $(ls $OUTPUT_DIR/traj | wc -l)"

If that grep comes up empty (CLI killed before post-eval ever ran), fall back to find /tmp -maxdepth 7 -name "${RUN_ID}_${BENCHMARK}" -type d 2>/dev/null | head — but note that find /tmp is slow (many dirs) and must be scoped to a reasonable depth. Use -maxdepth 7 and always direct stderr to /dev/null.

Decision tree — pick one path

1. Best path: just relaunch run_evals.sh with the same -r RUN_ID -b BENCHMARK. If all (or nearly all) trajectories + scores are already on blob, webeval's execution loop skips them, core.py's evaluate step loads cached scores, and post-eval runs naturally — producing a complete metrics.json (with easy/medium/hard difficulty splits, mean score, full histograms) on a fresh AML run. Uses one GPU for vLLM but no LLM spend and finishes in minutes. This is what you want when the original run was killed near 100% and you need the difficulty-split metrics that only metrics.json has — Phase 5b analyzers alone do not compute those.
2. Phase 5a (eval_only=true CLI) — same thing but without relaunching vLLM. Works when vLLM is still running from Phase 1. When it works, it writes to the original output dir and produces a full metrics.json.
3. Phase 5b (direct analyzer script) — bypasses the CLI entirely. Use when vLLM is already torn down, the box is busy, or you want zero chance of any verifier LLM call. Does NOT produce metrics.json and does NOT produce difficulty splits — only error-histogram and step-stat metrics. For AML dashboard parity, combine with Phase 5d to push those metrics onto the original AML run.
4. Phase 5d (resume-original-AML-run) — standalone fallback or combine with 5b. Recovers the original MLflow run_id from the AML workspace and attaches new metrics to that exact run instead of creating a second row on the dashboard.

Rule of thumb: if vLLM can still run (or can be restarted cheaply) and the GPU is free, prefer path 1. The full relaunch is the only path that gives you difficulty splits + metrics.json + Phase 5c ingest, all in one shot. Phase 5b+5d is the recovery-only path.

What eval_only=true actually does — read this before tweaking

Verified at webeval_next/src/webeval/core.py:303: when eval_only=true AND redo_eval=false (both default-path semantics), the "evaluate" step opens the existing scores/<eval_hash>.json, loads the saved score, and returns Stage.EVALUATED immediately — no LLM call, no verifier re-run, just a file read. The CLI then falls through to the post_eval_analyzers block in local.py:281, which re-aggregates the histograms and uploads them to AML via exp_logger.log_metrics(...).

Edge case worth calling out: a trajectory that executed but whose evaluator never ran (e.g., killed mid-evaluation) has no score.json on disk. For those, the CLI falls through past the early-return at core.py:327 and actually calls the verifier (the LLM). On a kill-near-100% this is typically <5% of tasks, and is usually desirable — those trajectories were going to stay unscored otherwise. But if the user wants a strict zero-verifier-runs pass, use the direct-script alternative below.

5a — default path: eval_only=true (reuses scores, only re-analyzes)

Mirror the original command's context (same run_id, benchmark, system, forwarded --extra-args). Keep model_url and metadata_endpoint pointing at the still-running vLLM — they're required for model_host.hash(), which is part of the output-folder path (runs/<system_hash>/<model_hash>/<user>/<bench>/<run_id>/). Without them the CLI computes a different path and analyzes an empty folder. Drop --start-vllm (vLLM is already running from Phase 1; don't spawn a second one) and do NOT pass --device-id (that would imply --start-vllm).

cd "$(git rev-parse --show-toplevel)/webeval_next"
# activate the webeval_next env (conda `webeval_next` or `webeval_next/.venv`)

# Read the running vLLM ports out of the launch log so we reuse the exact endpoints
# the original run used (auto-derived from DEVICE_ID: 5000+2*first_id / 5001+2*first_id).
VLLM_PORT=$(grep -oE "VLLM Port: [0-9]+" "$LAUNCH_LOG" | tail -1 | awk '{print $3}')
MODEL_PORT=$(grep -oE "Frontend Port: [0-9]+" "$LAUNCH_LOG" | tail -1 | awk '{print $3}')

python -m webeval.cli \
    benchmark=${BENCHMARK} \
    system=${SYSTEM} \
    run_id=${RUN_ID}_${BENCHMARK} \
    processes=${NUM_PROCESSES} \
    max_rounds=${MAX_ROUNDS} \
    eval_only=true \
    model_url=http://localhost:${VLLM_PORT}/v1/ \
    metadata_endpoint=http://localhost:${MODEL_PORT}/model \
    log_file=./logs/post_eval_${RUN_ID}_${BENCHMARK}.log \
    ${ORIG_EXTRA_ARGS} \
    2>&1 | tee -a "$LAUNCH_LOG"

Re-pass any system.extra_create_args.* and correct_tool_errors=true from the original $ARGUMENTS — some of them affect how scores get decoded / displayed. Leave redo_eval unset (default false).

5b — strict-zero-verifier-runs alternative

If a trajectory without a score.json should stay unscored (no verifier LLM calls under any circumstance), bypass the CLI entirely and call the analyzers directly. This logs to a new MLflow run rather than appending to the original AML run, so the dashboard will have a second row — note this trade-off in the report.

cd "$(git rev-parse --show-toplevel)/webeval_next"
# activate the webeval_next env (conda `webeval_next` or `webeval_next/.venv`)

python - <<'PY'
import os, sys, mlflow
from pathlib import Path
from webeval.post_eval_analyzers.analyzers import ErrorAnalyzer, StepStatsAnalyzer

# Resolve <output_folder>/traj/ — sibling of metrics.json, found by globbing
# the eval output dir for the run_id. run_evals.sh writes under the
# configured out_data_ref (AzureFolder uri); locally the Hydra config resolves
# this to /tmp/<...>/runs/<system_hash>/<model_hash>/<user>/<benchmark_hash>/<run_id>/
# If unsure, grep the original $LAUNCH_LOG for "output_folder" to find the exact path.
TRAJ_DIR = Path(os.environ["TRAJ_DIR"])
assert (TRAJ_DIR).is_dir() and TRAJ_DIR.name == "traj", f"pass TRAJ_DIR=<output>/traj, got {TRAJ_DIR}"

folders = [{"name": t, "files": list(t.iterdir())} for t in TRAJ_DIR.iterdir() if t.is_dir()]
print(f"analyzing {len(folders)} trajectories under {TRAJ_DIR}")

mlflow.set_experiment(os.environ.get("MLFLOW_EXPERIMENT", "eval-monitor-post-hoc"))
with mlflow.start_run(run_name=os.environ.get("RUN_NAME", TRAJ_DIR.parent.name)) as run:
    ErrorAnalyzer().analyze(folders, mlflow)
    StepStatsAnalyzer().analyze(folders, mlflow)
    print("MLflow run:", mlflow.get_tracking_uri(), run.info.run_id)
PY

The post_eval_report.txt is written next to traj/ either way.

Verification (both 5a and 5b)

• post_eval_report.txt should exist under <output_folder>/ (sibling of traj/). Print its first 40 lines to show the user the histograms.
• AML run URL (5a path only): the CLI prints Run URL: https://ml.azure.com/runs/... near the top. Grep $LAUNCH_LOG and echo it. For 5b, echo the local MLflow run URL instead, and flag that it's a new run (not the original).

If the re-analysis errors out (e.g., traj/ empty because vLLM never came up), surface the error plainly and do NOT claim the AML metrics were logged.

5d — resume the original AML run (attach metrics to the same row, not a new one)

Every webeval run starts an AML/MLflow run via AzureMLLogger.start() (aztool/aztool/mlflow/azureml_logger.py:37), which calls ws.start_run(experiment_name). When the CLI is killed in Phase 4, AzureMLLogger.stop() never runs, so the AML run stays in RUNNING status and any mlflow.end_run() later will reuse it if we supply the same run_id. That means Phase 5b's metrics can be pushed onto the original dashboard row instead of creating a second one.

Use this when Phase 5b is the only option (e.g., vLLM is gone, GPU is busy, you want zero verifier LLM calls) but you still want the metrics to land on the original ml.azure.com/runs/<run_id> URL.

# Prelude to Phase 5b (replaces the `mlflow.set_experiment(...)` line).
import mlflow
from aztool.workspace import Workspace, WorkspaceRef

ws = Workspace(workspace_ref=WorkspaceRef(
    resource_group="aifrontiers",
    subscription_id="d4fe558f-6660-4fe7-99ec-ae4716b5e03f",
    workspace_name="aifrontiers_ws"), tags={})
ws.connect_to_mlflow()

EXPERIMENT_ID = "151abaca-5a3a-46a2-88b8-7fb7f05d5134"  # osagent_eval

# Find the original run. AML rejects LIKE on tags; pull recent runs and
# client-side filter by run_id / run_name / start_time.
runs = mlflow.search_runs(
    experiment_ids=[EXPERIMENT_ID],
    max_results=200,
    order_by=["attributes.start_time DESC"],
)
# Filter by whatever you know — usually the run_name appears in dataframe,
# or you can match on approximate start_time from the launch log.
hits = runs[runs.apply(lambda r: "<your RUN_ID fragment>" in str(r.to_dict()), axis=1)]
orig_run_id = hits.iloc[0]["run_id"]
print("resuming", orig_run_id)

with mlflow.start_run(run_id=orig_run_id):
    ErrorAnalyzer().analyze(folders, mlflow)
    StepStatsAnalyzer().analyze(folders, mlflow)
    mlflow.set_tag("post_eval_source", "eval-monitor-phase5d")
    mlflow.set_tag("killed_near_100", "true")
    mlflow.set_tag("evaluated_over_total", f"{n_scored}/{n_total}")
    # exiting the context manager transitions the run RUNNING → FINISHED

Gotchas:

• The analyzers receive mlflow (the module) and call mlflow.log_metrics(...) on the active run. As long as you're inside a resumed start_run(run_id=...) context, they land on the original row.
• AML's MLflow REST rejects LIKE on tags. Use params.* filters (e.g., params.benchmark = 'om2w') or pull a recent batch and filter client-side on whatever string fragment identifies the run (RUN_ID, user alias, timestamp).
• This writes only the Phase 5b analyzer metrics (error histograms + step stats). It does not produce metrics.json or difficulty splits — those only exist when reduce_eval_results runs, which happens in paths 1 and 5a. If you need difficulty splits, prefer path 1 (full relaunch) over this.

Phase 5c — ingest into the unified ResultsStore

Post-eval covers the per-run AML dashboard (error histograms from ErrorAnalyzer), but the cross-run comparison dashboard is a separate, local-file store populated by webeval.results.ingest — which reads <output_folder>/<eval_hash>/metrics.json and writes an EvalRecord into ResultsStore. Without this step, the run never shows up on the scoreboard that compares checkpoints side-by-side.

Run only when the original command targeted a local or Azure-blob checkpoint (--model-url <local_path_or_blob_uri>). Skip for API-backed systems (gpt_solver*, anthropic_solver, operator_solver, fara_qwen3 without a --model-url) — those have no checkpoint identity to compare.

cd "$(git rev-parse --show-toplevel)/webeval_next"
# activate the webeval_next env (conda `webeval_next` or `webeval_next/.venv`)

CKPT_URL="<value of --model-url from original $ARGUMENTS>"
OUTPUT_DIR="<resolved output_folder — sibling of traj/ and post_eval_report.txt>"

python -m webeval.results.ingest \
    --output-dir "$OUTPUT_DIR" \
    --checkpoint-url "$CKPT_URL"
    # --checkpoint-name / --base-model auto-derived from URL if unset;
    # --tags '{"source":"eval-monitor","killed_near_100":"true"}' if the run was killed in Phase 4

Resolve OUTPUT_DIR by parsing the CLI stdout in Phase 5 for the run_id = runs/... line, or by globbing <out_data_ref>/runs/*/*/$USER/*/<RUN_ID>_<BENCHMARK>/ — the dir that contains traj/ plus an <eval_hash>/ holding metrics.json. If you pass --scan-dir instead of --output-dir, the tool will pick up every metrics.json under the root, which is useful when you don't know the exact path.

Report the ingest line (Ingested: benchmark=<name> score=<score>) to the user — that's the confirmation the scoreboard will pick it up on next refresh.

score=None is a silent ingest failure, not a success. If the output shows Ingested: benchmark=WebVoyager score=None, the ingest picked up the benchmark but never extracted the score — the scoreboard will show a blank cell. This happens when the benchmark's metrics.json uses a score key the ingest code doesn't look for (e.g., webvoyager's mean_score vs. the generic score key expected by webeval.results.ingest). Do not report this as a clean ingest. Surface it as a warning in Phase 7, along with:

• which key is actually in metrics.json (jq 'keys' <eval_hash>/metrics.json)
• which benchmark it is (so the user can fix the ingest code or file it as a known bug)

Phase 6 — optional --retry-blocked-on-complete

If this flag was present in Phase 0, after Phase 5 completes successfully, re-launch the same command with two mutations:

1. Add --browserbase (cloud-hosted browser with rotating residential IPs — bypasses cloudflare / anti-bot on sites like united.com, amazon.com, ign.com that commonly block datacenter IPs).
2. Replace --max-retries <N> with --max-retries $((N*2)) (or 10, whichever is larger).

Keep the same run_id. webeval's execution loop skips any task whose trajectory already has a final answer on disk, so the second run only re-executes the ones that hit TargetClosedError / browser timeouts / blocked-navigation failures. This is the cheap and correct way to rescue a run where 10-30% of trajectories failed due to IP-level site blocks.

Launch via nohup the same way as Phase 1, then repeat Phases 2–5 as a separate cycle. Label progress events with a [retry] prefix when reporting to the user so they know which pass they're reading.

Phase 7 — report

Verify deliverables BEFORE writing the report. "Exit code 0" and "post_eval_report.txt exists on disk" are necessary but not sufficient — the AML dashboard is the actual deliverable. Confirm it by querying MLflow directly:

cd "$(git rev-parse --show-toplevel)/webeval_next" && source .venv/bin/activate  # or conda activate webeval_next
python - <<PY
import mlflow
from aztool.workspace import Workspace, WorkspaceRef
ws = Workspace(workspace_ref=WorkspaceRef(
    resource_group="aifrontiers",
    subscription_id="d4fe558f-6660-4fe7-99ec-ae4716b5e03f",
    workspace_name="aifrontiers_ws"), tags={})
ws.connect_to_mlflow()
run_id = "$AML_RUN_ID"   # extracted from $LAUNCH_LOG via `grep -oE 'runs/[a-f0-9-]{36}'`
run = mlflow.get_run(run_id)
metrics = run.data.metrics
print(f"run status: {run.info.status}")
print(f"n_metrics logged: {len(metrics)}")
# Core smoke-test: at least mean_score + one ErrorAnalyzer bucket + one StepStats bucket.
required = ["mean_score"]
print("mean_score present:", "mean_score" in metrics)
print("any error bucket:", any(k.startswith("Long_Session") or k.startswith("Error_") for k in metrics))
print("any step-stat:", any(k.startswith("sstats_") for k in metrics))
PY

If any of those three checks come back false, the histograms didn't land — go back to Phase 5b+5d (don't write the success report yet).

On a clean finish:

• run_id, benchmark, system
• Final evaluated / started count and mean score (grep from <output_folder>/<eval_hash>/metrics.json)
• Path to post_eval_report.txt and the top 3 error buckets by count (grep the report)
• AML run URL plus the smoke-test confirmation above (mean_score=X, n_metrics=N, status=FINISHED)
• Phase 5c ingest line — flagged as warning if score=None
• Tail-along command hint: tail -F <EVAL_LOG> (so the user can keep reading after the run if they want)

On a stuck-kill:

• Above, plus: "killed at X/Y evaluated after T seconds; Phase 5 post-eval completed with error histograms logged to AML."
• Top 3 [fatal] / [log-error] events observed during the run.

On [mlflow-push-failing] seen during the run:

• Call out that the original AML run has no histograms (the one the CLI printed at launch) — supply its run_id anyway for auditability.
• Link to the recovery run that Phase 5d populated; note it was post-hoc.
• If the original run_id couldn't be recovered (e.g., run was deleted server-side), say so explicitly and point at the fresh Phase 5b run.

On --retry-blocked-on-complete:

• Both passes' scores side by side; delta in n_evaluated; delta in mean score.

Never claim success unless the MLflow smoke test above passed. A silent upload failure with exit code 0 is the single most common way this skill has misreported success — always verify server-side.

Encoded failure modes

Appendix — why the eval_only=true re-invocation works

webeval_next/src/webeval/environments/local.py::LocalEnvironment.run() (around line 281) unconditionally runs every post_eval_analyzers entry after run_eval_multiple_examples_with_progress returns. When eval_only=true, the inner loop short-circuits execution for every task whose trajectory is already on disk but still emits a results list that feeds reduce_eval_results → exp_logger.log_metrics(mlflow_compat_metrics) → ErrorAnalyzer.analyze(...) → histogram upload.

configs/environments/local.yaml wires ErrorAnalyzer + StepStatsAnalyzer by default, so any benchmark run picks them up automatically. The AML dashboard's error histograms (ParseToolCallError, ActionExecutionError, BadRequestError (Exceeded Token Budget), …) are dimensionally stable across runs because DEFAULT_ERROR_CLASSES_TO_SCAN in post_eval_analysis.py always emits a row per class even if count=0.