ai_agents

Imported from zhannn668/seeed-respeaker-agora-tenframework (ai_agents/AGENTS.md). Install upstream with npx skills add zhannn668/seeed-respeaker-agora-tenframework --skill ai_agents. Copyright stays with the author.

AGENTS.md

This file provides guidance to AI coding assistants when working with code in this repository.

Repository Overview

This is the TEN Framework AI Agents repository, a modular platform for building real-time AI agents with voice, video, and multimodal capabilities. The framework uses a graph-based architecture where extensions (ASR, LLM, TTS, RTC, tools) are connected via property.json configurations to create complete AI agent pipelines.

Development Guidelines

Do Not Modify Git-Ignored Files

IMPORTANT: Do not modify files that are ignored by git - they are automatically generated or managed by build tools.

Common auto-generated files in this repository (see agents/.gitignore):

• manifest-lock.json - Generated by tman during dependency resolution
• compile_commands.json - Generated by build system
• BUILD.gn - Generated build configuration
• .gn, .gnfiles - Generated build system links
• out/ - Build output directory
• .ten/ - TEN runtime generated files
• bin/main, bin/worker - Compiled binaries
• .release/ - Release packaging output
• *.log files - Runtime logs
• build/ directories - Build artifacts
• node_modules/ - JavaScript dependencies

When making changes:

• Focus on source files: *.py, *.go, *.ts, *.tsx, manifest.json, property.json, Taskfile.yml
• Let build tools regenerate their output files
• If you need to modify build behavior, edit the source configuration (e.g., Taskfile.yml, package.json) rather than generated files

YAGNI (You Ain't Gonna Need It)

Only implement functionality when it is actually needed, not when you foresee that it might be needed:

• Don't add features, configuration options, or abstractions for hypothetical future requirements
• Don't create helper functions or utilities for one-time operations
• Don't add extra parameters "just in case" they might be useful later
• If you need it later, add it later - when you understand the actual requirements

KISS (Keep It Simple, Stupid)

Prefer simple, straightforward solutions over complex ones:

• Write code that is easy to read and understand
• Avoid over-engineering - choose the simplest approach that solves the problem
• Don't add unnecessary layers of abstraction
• Three similar lines of code is often better than a premature abstraction
• Only add error handling and validation where genuinely needed (system boundaries, user input, external APIs)
• Avoid backwards-compatibility hacks - if something is unused, delete it completely

Core Architecture

Extension System

The TEN Framework is built around extensions - modular components that provide specific capabilities (ASR, TTS, LLM, tools, etc.). Extensions communicate through a graph-based message passing system.

Extension Structure:

• manifest.json - Extension metadata, dependencies, and API interface definitions
• property.json - Default configuration and parameters (supports ${env:VAR_NAME} syntax)
• addon.py - Registration class using @register_addon_as_extension decorator
• extension.py - Main extension logic, inheriting from base classes like AsyncASRBaseExtension, AsyncTTSBaseExtension, etc.
• tests/ - Standalone test directory with bin/start script

Base Extension Classes:

• Located in agents/ten_packages/system/ten_ai_base/interface/ten_ai_base/
• Common bases: AsyncASRBaseExtension, AsyncTTSBaseExtension, LLMBaseExtension
• API interfaces defined in agents/ten_packages/system/ten_ai_base/api/*.json

Graph-Based Configuration

Agents are configured using predefined_graphs in property.json:

{
  "ten": {
    "predefined_graphs": [{
      "name": "voice_assistant",
      "auto_start": true,
      "graph": {
        "nodes": [
          {"name": "stt", "addon": "deepgram_asr_python", "property": {...}},
          {"name": "llm", "addon": "openai_llm2_python", "property": {...}},
          {"name": "tts", "addon": "elevenlabs_tts2_python", "property": {...}}
        ],
        "connections": [
          {
            "extension": "main_control",
            "data": [{"name": "asr_result", "source": [{"extension": "stt"}]}]
          }
        ]
      }
    }]
  }
}

Connection Types:

• data - Data messages (asr_result, text, etc.)
• cmd - Command messages (on_user_joined, tool_register, etc.)
• audio_frame - Audio stream data (pcm_frame)
• video_frame - Video stream data

Repository Structure

agents/
├── ten_packages/
│   ├── extension/          # 60+ extensions (ASR, TTS, LLM, tools)
│   │   ├── deepgram_asr_python/
│   │   ├── openai_llm2_python/
│   │   ├── elevenlabs_tts2_python/
│   │   └── ...
│   ├── system/             # Core framework packages
│   │   ├── ten_ai_base/    # Base classes and API interfaces
│   │   ├── ten_runtime_python/
│   │   └── ten_runtime_go/
│   └── addon_loader/       # Language-specific addon loaders
├── examples/               # Complete agent examples
│   ├── voice-assistant/    # Basic voice agent (STT→LLM→TTS)
│   ├── voice-assistant-realtime/  # OpenAI Realtime API
│   ├── voice-assistant-video/     # Vision capabilities
│   └── ...
├── integration_tests/
│   ├── asr_guarder/        # ASR integration testing framework
│   └── tts_guarder/        # TTS integration testing framework
├── scripts/                # Build and package scripts
└── manifest.json           # App-level manifest

server/                     # Go API server
├── main.go                 # HTTP server for agent lifecycle
└── internal/               # Server implementation

playground/                 # Next.js frontend UI
└── src/                    # React components

esp32-client/              # ESP32 hardware client

Development Commands

Build & Lint

# Lint all Python extensions
task lint

# Lint specific extension
task lint-extension EXTENSION=deepgram_asr_python

# Format Python code with black
task format

# Check code formatting
task check

Testing

# Run all tests (server + extensions)
task test

# Test specific extension
task test-extension EXTENSION=agents/ten_packages/extension/elevenlabs_tts_python

# Test extension without reinstalling dependencies
task test-extension-no-install EXTENSION=agents/ten_packages/extension/elevenlabs_tts_python

# Run ASR guarder integration tests
task asr-guarder-test EXTENSION=azure_asr_python CONFIG_DIR=tests/configs

# Run TTS guarder integration tests
task tts-guarder-test EXTENSION=bytedance_tts_duplex CONFIG_DIR=tests/configs

Extension Test Runner: Extensions with tests/ directories use standalone testing:

• Test entry point: tests/bin/start (shell script)
• Sets PYTHONPATH to include ten_runtime_python and ten_ai_base interfaces
• Runs pytest or custom test harness
• Requires .env file with API keys

Running Examples

Each example has its own Taskfile.yml:

# Navigate to example directory
cd agents/examples/voice-assistant

# Install dependencies (frontend, server, tenapp)
task install

# Run everything (API server, frontend, TMAN Designer)
task run

# Run individual components
task run-api-server
task run-frontend
task run-gd-server     # TMAN Designer on port 49483

Ports:

• Frontend: http://localhost:3000
• API Server: http://localhost:8080
• TMAN Designer: http://localhost:49483

Server API

The Go server manages agent processes via REST API:

POST /start - Start an agent with a graph

{
  "request_id": "uuid",
  "channel_name": "test_channel",
  "user_uid": 176573,
  "graph_name": "voice_assistant",
  "properties": {},
  "timeout": 60
}

POST /stop - Stop an agent POST /ping - Keep agent alive (if timeout != -1)

Python Development

PYTHONPATH Configuration

Extensions require specific PYTHONPATH to import TEN runtime and AI base:

export PYTHONPATH="./agents/ten_packages/system/ten_runtime_python/lib:./agents/ten_packages/system/ten_runtime_python/interface:./agents/ten_packages/system/ten_ai_base/interface"

This is configured in:

• Taskfile.yml tasks (lint, test-extension)
• pyrightconfig.json (per-example executionEnvironments)
• Extension test scripts (tests/bin/start)

Type Checking

Pyright is configured in pyrightconfig.json:

• Mode: basic
• Key checks: reportUnusedCoroutine, reportMissingAwait, reportUnawaitedAsyncFunctions = error
• Most type checks disabled to accommodate dynamic TEN runtime APIs
• Separate execution environments per example to resolve imports correctly

Extension Development Patterns

Creating a New Extension

1. Inherit from base class:

   from ten_ai_base.asr import AsyncASRBaseExtension
   
   class MyASRExtension(AsyncASRBaseExtension):
       async def on_init(self, ten_env: AsyncTenEnv) -> None:
           await super().on_init(ten_env)
           # Load config from property.json
           config_json, _ = await ten_env.get_property_to_json("")

2. Register addon:

   from ten_runtime import Addon, register_addon_as_extension
   
   @register_addon_as_extension("my_asr_python")
   class MyASRExtensionAddon(Addon):
       def on_create_instance(self, ten: TenEnv, addon_name: str, context) -> None:
           ten.on_create_instance_done(MyASRExtension(addon_name), context)

3. Define API interface in manifest.json:

   {
     "api": {
       "interface": [
         {"import_uri": "../../system/ten_ai_base/api/asr-interface.json"}
       ]
     }
   }

Common Patterns

Config with Pydantic: Extensions typically use Pydantic models for config validation:

from pydantic import BaseModel

class MyConfig(BaseModel):
    api_key: str
    model: str = "default"

Environment Variables: Use ${env:VAR_NAME} or ${env:VAR_NAME|} (with fallback) in property.json:

{"api_key": "${env:DEEPGRAM_API_KEY}"}

Logging Categories:

• LOG_CATEGORY_KEY_POINT - Important lifecycle events
• LOG_CATEGORY_VENDOR - Vendor-specific status/errors

Message Sending:

• await self.send_asr_result(asr_result)
• await self.send_asr_error(module_error, vendor_info)
• await self.send_asr_finalize_end()

Configuration and Params Handling

Params Dict Pattern: Extensions using HTTP-based services (TTS, ASR, etc.) typically store all configuration in a params dictionary that gets passed to the vendor API. Follow these patterns:

1. Sensitive Parameters (API Keys):

• Store api_key inside params dict in property.json and config
• Extract for authentication headers in the client constructor
• Strip from params only when creating the HTTP request payload (not during config processing)

Example from rime_http_tts:

# config.py - Keep api_key in params throughout
class RimeTTSConfig(AsyncTTS2HttpConfig):
    params: dict[str, Any] = Field(default_factory=dict)

    def validate(self) -> None:
        if "api_key" not in self.params or not self.params["api_key"]:
            raise ValueError("API key is required")

# rime_tts.py - Extract for headers, strip from payload
class RimeTTSClient(AsyncTTS2HttpClient):
    def __init__(self, config: RimeTTSConfig, ten_env: AsyncTenEnv):
        self.api_key = config.params.get("api_key", "")
        self.headers = {"Authorization": f"Bearer {self.api_key}"}

    async def get(self, text: str, request_id: str):
        # Shallow copy and strip api_key before sending
        payload = {**self.config.params}
        payload.pop("api_key", None)

        async with self.client.stream("POST", url, json={"text": text, **payload}):
            # ...

2. Parameter Type Preservation:

• Define param types correctly in manifest.json api.property.properties
• Use "int32" for integer params, "float64" for floats, "string" for strings
• Pydantic will coerce types based on manifest.json schema definitions

Example:

// manifest.json
"params": {
  "type": "object",
  "properties": {
    "api_key": {"type": "string"},
    "top_p": {"type": "int32"},      // Integer param
    "temperature": {"type": "float64"},  // Float param
    "samplingRate": {"type": "int32"}
  }
}

3. Param Transformations in update_params():

• Add vendor-required params (e.g., audioFormat, segment)
• Normalize alternative key names (e.g., sampling_rate → samplingRate)
• Remove internal-only params from blacklist
• DO NOT strip api_key here (strip only when making requests)

Example:

def update_params(self) -> None:
    # Add required params
    self.params["audioFormat"] = "pcm"
    self.params["segment"] = "immediate"

    # Normalize keys
    if "sampling_rate" in self.params:
        self.params["samplingRate"] = int(self.params["sampling_rate"])
        del self.params["sampling_rate"]

    # Remove internal-only params (NOT api_key)
    blacklist = ["text"]
    for key in blacklist:
        self.params.pop(key, None)

4. Sensitive Data in Logging:

• Implement to_str() method that encrypts sensitive fields
• Use utils.encrypt() for api_key before logging

Example:

def to_str(self, sensitive_handling: bool = True) -> str:
    if not sensitive_handling:
        return f"{self}"

    config = copy.deepcopy(self)
    if config.params and "api_key" in config.params:
        config.params["api_key"] = utils.encrypt(config.params["api_key"])
    return f"{config}"

Advanced Extension Patterns

1. Bidirectional Extensions (Input + Output)

Extensions can both receive from and send to the TEN graph. This pattern is useful for bridges, proxies, and transport layers.

Key techniques:

• Store self.ten_env reference in __init__ for use in callbacks
• Implement on_audio_frame() or on_data() to receive from graph
• Use callbacks to bridge external systems with TEN graph

Example:

class MyExtension(AsyncExtension):
    def __init__(self, name: str):
        super().__init__(name)
        self.ten_env: AsyncTenEnv = None  # Store for callbacks

    async def on_init(self, ten_env: AsyncTenEnv):
        self.ten_env = ten_env  # Save reference

    async def on_audio_frame(self, ten_env, audio_frame):
        # Receive from graph → forward to external system
        buf = audio_frame.lock_buf()
        pcm_data = bytes(buf)
        audio_frame.unlock_buf(buf)
        self.external_system.send(pcm_data)

    async def _external_callback(self, data):
        # Receive from external system → send to graph
        audio_frame = AudioFrame.create("pcm_frame")
        # ... configure frame ...
        await self.ten_env.send_audio_frame(audio_frame)

2. AudioFrame Creation Pattern

Standard pattern for creating and sending AudioFrames:

# Create frame
audio_frame = AudioFrame.create("pcm_frame")

# Set properties (order matters - do before alloc_buf)
audio_frame.set_sample_rate(16000)
audio_frame.set_bytes_per_sample(2)  # 16-bit audio
audio_frame.set_number_of_channels(1)  # mono
audio_frame.set_data_fmt(AudioFrameDataFmt.INTERLEAVE)
audio_frame.set_samples_per_channel(len(pcm_data) // 2)

# Allocate and fill buffer
audio_frame.alloc_buf(len(pcm_data))
buf = audio_frame.lock_buf()
buf[:] = pcm_data
audio_frame.unlock_buf(buf)

# Send to graph
await ten_env.send_audio_frame(audio_frame)

Key points:

• Always use AudioFrame.create() factory method (not __init__)
• Set all properties before calling alloc_buf()
• Lock/unlock buffer pattern ensures thread safety
• Calculate samples: samples_per_channel = total_bytes / (bytes_per_sample * channels)

TMAN Tool

tman is the TEN package manager used for:

• tman install - Install extension dependencies from manifest.json
• tman run start - Run the tenapp
• tman designer - Start TMAN Designer (visual graph editor)

Integration Tests

ASR Guarder (agents/integration_tests/asr_guarder/):

• Tests ASR extensions with audio streams
• Validates reconnection, finalization, multi-language, metrics
• Uses pytest fixtures and conftest.py

TTS Guarder (agents/integration_tests/tts_guarder/):

• Tests TTS extensions with text input
• Validates flush, corner cases, invalid text handling, metrics

Both use template-based manifest generation:

sed "s/{{extension_name}}/$EXT_NAME/g" manifest-tmpl.json > manifest.json

Environment Configuration

Required .env variables depend on extensions used. Common ones:

RTC:

• AGORA_APP_ID, AGORA_APP_CERTIFICATE

LLM:

• OPENAI_API_KEY, OPENAI_MODEL, OPENAI_API_BASE
• AZURE_OPENAI_REALTIME_API_KEY, AZURE_OPENAI_REALTIME_BASE_URI

ASR:

• DEEPGRAM_API_KEY, AZURE_ASR_API_KEY, AZURE_ASR_REGION

TTS:

• ELEVENLABS_TTS_KEY, AZURE_TTS_KEY, AZURE_TTS_REGION

See .env.example for complete list.

Key Files to Check

When working on:

• New extension → Check agents/ten_packages/extension/<similar_extension>/ for patterns
• API changes → Check agents/ten_packages/system/ten_ai_base/api/*.json
• Graph config → Check agents/examples/*/tenapp/property.json
• Test setup → Check agents/ten_packages/extension/*/tests/bin/start

Common Issues

Import errors in extensions:

• Ensure PYTHONPATH includes ten_runtime_python and ten_ai_base interfaces
• Check pyrightconfig.json executionEnvironments for the example

Extension not loading:

• Verify manifest.json dependencies match installed packages
• Check addon.py decorator name matches property.json "addon" field
• Run tman install in the tenapp directory

Test failures:

• Ensure .env has required API keys
• Check PYTHONPATH in tests/bin/start script
• Verify test configs in tests/configs/ directory