Imported from zhuabo001/skill-evaluation-system (
AGENTS.md). Install upstream withnpx skills add zhuabo001/skill-evaluation-system. Copyright stays with the author.
[skill-eval-github] recent context, 2026-06-18 2:53pm GMT+8
Legend: 🎯session 🔴bugfix 🟣feature 🔄refactor ✅change 🔵discovery ⚖️decision Format: ID TIME TYPE TITLE Fetch details: get_observations([IDs]) | Search: mem-search skill
Stats: 26 obs (5,181t read) | 192,592t work | 97% savings
Jun 10, 2026
531 3:38p 🔵 Eval test grade loop document is 719 lines 532 " 🔵 Eval-test-grade-loop document details Skill Creator full workflow 533 " 🔵 Full grading and improvement pipeline documented in 719-line reference 534 3:39p 🔵 Skill Creator reference implementation has 5307 lines across 15 files 535 " 🔵 Skill Creator schemas define 8 JSON data contracts 536 3:42p 🔵 Grader agent uses strict evidence-based assertion grading with eval critique 537 " 🔵 Blind comparator uses 1-5 rubric scoring across content and structure dimensions 538 " 🔵 Post-hoc analyzer bridges blind comparison with actionable skill improvements 539 " 🔵 Python scripts architecture revealed — claude -p subprocess pattern 540 3:43p 🔵 run_eval.py detects skill triggering via stream event monitoring 541 " 🔵 run_loop.py orchestrates automated description optimization with train/test split 542 " 🔵 improve_description.py uses claude -p subprocess for iterative prompt engineering 543 " 🔵 aggregate_benchmark.py supports two directory layouts and validates field names 544 " 🔵 quick_validate.py enforces strict frontmatter validation rules 545 " 🔵 package_skill.py validates before packaging and supports exclusion patterns 546 " 🔵 utils.py parses SKILL.md frontmatter with YAML multiline support 547 3:49p 🔵 generate_review.py serves zero-dependency HTML review viewer with two modes 548 " 🔵 viewer.html implements single-page eval review with benchmark tab 549 " 🔵 eval_review.html provides interactive eval set editing UI 550 4:10p ⚖️ Desktop client architecture decisions 551 4:31p ✅ Skill evaluation project directory structure explored 552 4:32p 🔵 AGENTS.md found but appears empty While exploring the skill-eval-github repository, the primary session discovered that AGENTS.md exists but contains no substantive content when read via sed. This suggests the file may be empty or contain only whitespace, despite being present. The reference/ directory is also empty at shallow depth. A .git history exists, indicated by COMMIT_EDITMSG. ~181t 🔍 307
553 4:45p ⚖️ Skill Studio MVP blueprint created with Tauri-based architecture A comprehensive MVP blueprint (skill-studio-mvp-blueprint.md) was created for Skill Studio, a local Tauri desktop application for engineering AI skills. The document spans 18 sections covering product positioning, architecture, data models, permission model, review UI, exporters, reference asset migration plan, MVP milestones, and V2 roadmap. The architecture defines four layers: Tauri Desktop Shell (cross-platform shell, file picker, permission dialogs), Core Engine (model-agnostic business logic for scheduling, grading, benchmarking), Agent Runtime (controlled execution environment with tool protocol and transcript recording), and Model Adapters (Anthropic/OpenAI unified interface). The MVP scope is carefully scoped to exclude auto-improvement, team features, remote runners, and cloud benchmarking — reserving those for V2 while ensuring the data models and interfaces accommodate future expansion. ~719t ⚖️ 17,511
Jun 16, 2026
577 10:42p 🔵 Skill Studio MVP comprehensive specification created The primary session designed a comprehensive product specification for "Skill Studio" - a local desktop application for skill development, evaluation, and iteration. The specification establishes a Tauri-based desktop application architecture prioritizing a headless vertical slice approach: schemas/artifacts → NativeAgentRuntime → with_skill+baseline execution → grader+benchmark → Tauri UI → review UI polish → exporters.
Key architectural decisions include using Tauri instead of Electron to reduce resource overhead, implementing a NativeAgentRuntime rather than depending on agent frameworks like DeepAgents, and adopting a workspace + permission prompt security model instead of container-level isolation. The evaluation model requires dual execution (with_skill injecting skill instructions vs baseline without) to generate comparable artifacts for human review.
The project defines clear artifact contracts for skill projects (skill.yaml, instructions.md), evaluation sets (evals/evals.json), run results (transcripts, metrics, timing, grading), and benchmark aggregation. The development routine specifies 8 phases from engineering skeleton through exporters, with recommended starting point being schema definition, core project loading, and minimal runtime execution before UI development.
~628t -
578 10:45p ✅ V1 Phase 0 engineering skeleton plan created The primary session created a detailed Phase 0 implementation plan that translates the high-level V1 development routine into concrete actionable tasks. The plan establishes a monorepo structure with clear package boundaries: schemas provide shared data contracts, core handles business lifecycle, runtime implements NativeAgentRuntime, model-adapters handle provider API adaptation, and document-tools provide file processing helpers.
Phase 0 focuses on engineering infrastructure rather than feature implementation, with 7 specific tasks covering workspace configuration, package skeletons, shared type definitions, interface drafts, sample fixtures, desktop app placeholder, and foundation test commands. The plan includes a realistic sample skill project fixture for a file transformation scenario to support subsequent phases.
Technical stack decisions specify TypeScript for package implementations, Rust for Tauri backend integration, and Python for document processing through subprocess calls. The plan includes risk checkpoints to prevent premature UI development, over-abstraction, delayed schema definition, and toy-like fixtures that wouldn't expose real implementation challenges.
~603t -
579 " ✅ Detailed Phase 0 implementation plan completed in worktree Working in a git worktree environment, the primary session completed a comprehensive Phase 0 implementation plan that translates the high-level V1 development routine into actionable engineering tasks. The 494-line plan provides detailed guidance for establishing the monorepo infrastructure needed to support subsequent development phases.
The plan establishes clear package boundaries and responsibilities, specifying that schemas provide shared data contracts, core handles business lifecycle operations, runtime implements NativeAgentRuntime, model-adapters handle provider API differences, and document-tools provide file processing capabilities. Technical stack decisions designate TypeScript for package implementations, Rust for Tauri backend integration, and Python for document processing through subprocess calls.
Seven concrete tasks are defined with specific deliverables and acceptance criteria, covering workspace configuration, package skeleton creation, shared type definitions, interface drafts, sample fixture creation, desktop app placeholder, and foundation test commands. The plan includes risk mitigation strategies for common pitfalls like premature UI development, over-abstraction, delayed schema definition, and insufficient test fixtures.
~636t -
Access 193k tokens of past work via get_observations([IDs]) or mem-search skill.
