gemini-live

Imported from manishkjs/gemini_live_pipecat (docs-site/public/gemini-live/SKILL.md). Install upstream with npx skills add manishkjs/gemini_live_pipecat --skill gemini-live. Copyright stays with the author.

Gemini Live API & Pipecat Production Engineering Reference (Complete 25-Section Handbook)

Installation: Save this file to .agents/skills/gemini-live/SKILL.md or ~/.gemini/config/skills/gemini-live/SKILL.md so your coding agent automatically enforces all 25 Gemini Live protocol, audio, tool-calling, and token-optimization invariants.

1. Protocol, Endpoints, Model Compatibility & Framing Matrix

Protocol, Framing & Model Compatibility Invariants (Gemini 2.5 vs. Gemini 3.1 Live)

When building or migrating between gemini-live-2.5-flash-native-audio and gemini-3.1-flash-live-preview, enforce these seven protocol invariants:

1. response_modalities Must Be Singleton ["AUDIO"]:
   • Never pass ["AUDIO", "TEXT"] in LiveConnectConfig. Requesting dual modalities triggers an immediate WebSocket close with 1007 Invalid Argument. Retrieve text transcripts via input_audio_transcription and output_audio_transcription.
2. proactive_audio & enable_affective_dialog Compatibility Guard:
   • gemini-live-2.5-flash-native-audio supports proactive_audio=True and enable_affective_dialog=True.
   • Setting either flag on gemini-3.1-flash-live-preview triggers a 1007 WebSocket handshake failure. Gate both parameters behind "2.5" in model_id.
3. Wire Format Changes (realtime_input vs. client_content):
   • On Gemini 3.1 Live, send_realtime_input(media=...) is deprecated in favor of send_realtime_input(audio=Blob(data=pcm, mime_type="audio/pcm;rate=16000")) and send_realtime_input(text=...).
   • send_client_content on Gemini 3.1 Live only supports appending turns (turn_complete=True or turn_complete=False) and cannot replace prior conversation history mid-session.
4. Thinking Configuration (thinking_level vs. thinking_budget):
   • Gemini 2.5 Flash Native Audio: Accepts ThinkingConfig(thinking_budget=0, include_thoughts=False) to disable thinking completely.
   • Gemini 3.1 Flash Live: Uses discrete ThinkingLevel ("MINIMAL", "LOW", "MEDIUM", "HIGH"). Never pass thinking_budget=0 to Gemini 3.1 Live; it degrades multi-step tool parameter copying and slot accuracy in multi-turn tool workflows. Always configure ThinkingConfig(thinking_level="MINIMAL", include_thoughts=False).
5. S2ST Interpreter Mode vs. Conversational Agent Mode:
   • Do not combine TranslationConfig (continuous speech-to-speech simultaneous interpretation) with conversational system instructions or function calling. Keep unidirectional S2ST interpreter pipelines and interactive bidirectional voice agents strictly separate.
6. Session Lifecycle Rotation (10 to 15 min):
   • The server rotates WebSocket connections after 10 to 15 minutes with close code 1008 after emitting a session_resumption_update event. Reconnect automatically using session_resumption_handle=new_handle.
7. Audio Formats & Setup Handshake ACK:
   • Input Audio (User to Gemini): PCM 16kHz 16-bit little-endian mono (audio/pcm;rate=16000).
   • Output Audio (Gemini to Client): PCM 24kHz 16-bit little-endian mono (audio/pcm;rate=24000).
   • Setup Handshake ACK: Vertex AI emits {"setupComplete": {"sessionId": "<UUID>"}} before streaming audio.

2. Speech-to-Text Routing & Cascaded Model Hierarchy

A. Speech-to-Text (STT) & Reasoning Config Invariants

• Live Transcription (input_audio_transcription & output_audio_transcription):
  • Enable both input_audio_transcription=AudioTranscriptionConfig() and output_audio_transcription=AudioTranscriptionConfig() inside LiveConnectConfig so your backend receives real-time user and assistant transcripts alongside native audio streams.
  • AI Studio AudioTranscriptionConfig Invariant: The language_codes field inside AudioTranscriptionConfig is only supported on Vertex AI (USE_VERTEXAI=true). Passing language_codes in Google AI Studio mode (GEMINI_API_KEY) raises: ValueError: language_codes parameter is only supported in Vertex AI mode, not in Gemini Developer API mode. When operating in AI Studio mode, instantiate AudioTranscriptionConfig() with zero arguments.
  • Model-Generation Thinking Config Invariant (thinking_budget=0 vs. ThinkingLevel.MINIMAL):

    from google.genai import types
    
    if "2.5" in model_id:
        thinking_cfg = types.ThinkingConfig(thinking_budget=0, include_thoughts=False)
    else:
        level = getattr(types.ThinkingLevel, "MINIMAL", "minimal")
        thinking_cfg = types.ThinkingConfig(thinking_level=level, include_thoughts=False)

  • Pipecat 1.2+ LLMUserAggregator Turn Invariant: LLMUserAggregator ignores InterimTranscriptionFrame. Custom STT services must emit TranscriptionFrame and call _handle_transcription(transcript, is_final=True) to commit speech text into the LLM turn context.
  • Pipecat 1.2+ VAD Config Invariant: FastAPIWebsocketParams does not accept vad_analyzer (passing it raises a TypeError). Connect VADProcessor directly inside the frame pipeline.
• chirp_3 (Cloud Speech-to-Text v2 Multilingual Sidecar):
  • Location Invariant: Must use location="us" (US Multi-Region) with regional endpoint us-speech.googleapis.com. Calling us-central1 fails with 400 Expected resource location to be us.
  • Languages: Pass language_codes=["en-US", "hi-IN"] for simultaneous multilingual recognition.
  • Streaming: Set enable_interim_results=True for low-latency partial transcripts.
• chirp_2: Hosted in location="us-central1" (us-central1-speech.googleapis.com).
• latest_long / latest_short / telephony: Available in us-central1 and global.

B. Cascaded Model Tier Standards (STT -> LLM -> TTS Alternative Pipeline)

When operating a modular cascaded pipeline alongside native Speech-to-Speech:

• Default STT: Cloud Speech-to-Text v2 (chirp_3 in location="us") or Gemini Live STT (response_modalities=["TEXT"]).
• Default LLM: gemini-2.5-flash-lite or gemini-2.5-flash (thinking_budget=0 for low-latency dialogue).
• Default TTS: Cloud Text-to-Speech chirp_3 HD voices or Gemini TTS (gemini-2.5-flash-preview-tts).

3. IAM Permissions & Local Dev Tunneling

Required Roles by Service

Grant the runtime service account the following least-privilege IAM roles:

• Vertex AI Live & LLMs: roles/aiplatform.user (aiplatform.endpoints.predict).
• Speech-to-Text v2: roles/speech.client (speech.recognizers.recognize).
• Text-to-Speech: roles/texttospeech.client (texttospeech.synthesize).
• Session / Memory Storage (Optional): roles/datastore.user or roles/storage.objectAdmin.
• ADC Quota Project: Local developer credentials require gcloud auth application-default set-quota-project <YOUR_GCP_PROJECT_ID>.

SSH Tunneling Invariant (Remote Linux VM -> Local Browser)

• Avoid HTTP reverse proxies that strip WebSocket Upgrade headers or block wss:// audio streams.
• Forward the application port over SSH so your local browser connects to localhost:

  ssh -L 7860:localhost:7860 user@your-dev-vm.example.com

  Open http://localhost:7860/ in Chrome, which treats localhost as a secure context (window.isSecureContext === true) for navigator.mediaDevices.getUserMedia() microphone capture.

4. Duplex Voice Pipeline, Idle Routing & Silero VAD Architecture

A. Silero VAD Processor Invariant (Pipecat 1.2+)

• FastAPIWebsocketParams Limitation: In Pipecat 1.2+, WebSocket transports (FastAPIWebsocketTransport) do not run VAD internally. Passing vad_analyzer to FastAPIWebsocketParams is ignored or raises a TypeError.
• Canonical VAD Pipeline Pattern:
  1. Instantiate SileroVADAnalyzer with conversational parameters:

     from pipecat.audio.vad.silero import SileroVADAnalyzer
     from pipecat.audio.vad.vad_analyzer import VADParams
     from pipecat.processors.audio.vad_processor import VADProcessor
     
     vad_analyzer = SileroVADAnalyzer(
         params=VADParams(
             confidence=0.7,      # Strict speech probability threshold (filters breaths/clicks)
             start_secs=0.2,      # 200ms speech onset confirmation
             stop_secs=0.4,       # 400ms natural conversational pause threshold
             min_volume=0.6,      # Minimum volume threshold to ignore ambient room noise
         )
     )
     vad_processor = VADProcessor(vad_analyzer=vad_analyzer)

  2. Insert vad_processor immediately after start_trigger in pipeline_elements to emit VADUserStartedSpeakingFrame and VADUserStoppedSpeakingFrame downstream:

     pipeline_elements = [
         transport.input(),
         start_trigger,
         vad_processor,
         stt,
         TranscriptionBroadcaster(participant="User"),
         context_aggregator.user(),
         llm,
         TranscriptionBroadcaster(participant="Bot"),
         tts,
         context_aggregator.assistant(),
         transport.output()
     ]

  3. Pass vad_analyzer to LLMUserAggregatorParams so turn-taking boundaries synchronize with LLM context aggregation:

     user_params = LLMUserAggregatorParams(vad_analyzer=vad_analyzer)
     context_aggregator = LLMContextAggregatorPair(context, user_params=user_params)

B. Greeting Turn Invariant (Anti-Duplicate Audio)

• In StartTriggerProcessor.process_frame(), push ONLY LLMRunFrame().
• NEVER push both LLMContextFrame and LLMRunFrame(). LLMUserAggregator emits context automatically upon receiving LLMRunFrame(); pushing both triggers duplicate parallel greeting speech.

C. Pipecat Upstream Frame Routing & User Idle Management

• Upstream Silence Blindness Hazard: Pipecat transports push audio downstream to the LLM, and the LLM emits TranscriptionFrame, UserStartedSpeakingFrame, and InterruptionFrame downstream. Any FrameProcessor (such as UserIdleProcessor) placed upstream of the LLM never sees downstream transcription frames. A short idle timeout (such as 5s) will mistakenly treat active user speech as silence and disconnect the call.
• The Solution: Wire direct activity reporting from the LLM service to the idle processor:

  # agent_live.py
  if hasattr(self, "user_idle_processor") and self.user_idle_processor:
      self.user_idle_processor.record_activity("speech_transcription")

• Cadence Guidelines: Use a 15s idle timer with check-in prompts at 15s, 30s, and 45s, disconnecting only after 60s of uninterrupted silence.

D. Pipecat WebSocket Receive Loop Invariant: Never Chain server_content Fields with elif

• The Terminal Frame Co-Occurrence Trap: In google-genai AsyncSession.receive(), Gemini Live frequently sends a terminal WebSocket frame where both message.server_content.model_turn (the final audio chunk) and message.server_content.turn_complete = True (or output_transcription) are populated on the same message.
• Anti-Pattern (elif Chain): Chaining if sc.interrupted: ... elif sc.model_turn: ... elif sc.turn_complete: ... elif sc.output_transcription: causes _handle_msg_turn_complete and _handle_msg_output_transcription to be skipped whenever model_turn is present on the final frame. Consequently, Pipecat never emits LLMFullResponseEndFrame for that turn, stalling LLMAssistantResponseAggregator and mis-bucketing per-turn usage rollups.
• Canonical Pattern (Independent if Blocks):

  sc = message.server_content
  if sc:
      if sc.interrupted:
          await host.broadcast_interruption()
      if sc.model_turn:
          await host._handle_msg_model_turn(message)
      if sc.input_transcription:
          await host._handle_msg_input_transcription(message)
      if sc.output_transcription:
          await host._handle_msg_output_transcription(message)
      if sc.grounding_metadata:
          await host._handle_msg_grounding_metadata(message)
      if sc.turn_complete:
          await host._handle_msg_turn_complete(message)
  if message.tool_call:
      await host._handle_msg_tool_call(message)

E. Pipecat ContextWindowCompressionConfig & SlidingWindow Wiring Invariant

• Silent No-Op Anti-Pattern: Passing a plain dictionary ({"enabled": True, "trigger_tokens": ...}) or storing target_tokens inside kwargs["extra"] on GeminiLiveInputParams is silently ignored by Pipecat, leaving long calls with unbounded audio and tool-result history.
• Canonical Pattern: Always construct a typed types.ContextWindowCompressionConfig with trigger_tokens=5000 (the Vertex AI minimum trigger floor) and types.SlidingWindow(target_tokens=3500):

  from google.genai import types
  
  if compression_enabled and trigger_tokens:
      target_tokens = target_tokens or 3500
      kwargs["context_window_compression"] = types.ContextWindowCompressionConfig(
          trigger_tokens=trigger_tokens,
          sliding_window=types.SlidingWindow(target_tokens=target_tokens),
      )

5. Live Observability & Diagnostic Buffer

• Zero-Disk In-Memory Ring Buffer: Intercept Loguru and Python standard logging into a bounded deque(maxlen=1500) in diagnostic_buffer.py, exposed via GET /api/logs and POST /api/logs/clear (polled every 1.5s by the debug UI).
• Token Fragment Filtering: Never log single-word streaming fragments:
  • User: Flush complete sentences upon end-of-sentence punctuation (., ।, ?, !) or VAD stop.
  • Bot: Accumulate streaming output chunks in _bot_turn_text_buffer; log the complete turn upon _handle_msg_turn_complete or _handle_msg_interrupted.

6. Latency Measurement & Telemetry Invariants

A. First-Packet Live TTFB (Gemini Live)

Calculate Time-to-First-Byte (TTFB) on whichever arrives first from the model turn (text transcription or PCM audio chunk):

if getattr(self, "_current_turn_ttft", None) is None and getattr(self, "_my_ttfb_start", None) is not None:
    self._current_turn_ttft = time.time() - self._my_ttfb_start
    self._my_ttfb_start = None

B. Speech-to-Text Latency & Turnaround Formulas

1. Cloud Speech v2 (CustomGoogleSTTService):

   • The 2ms Audio Stream Trap: Continuous raw audio frames flow every 20ms during silence. Calculating now - last_audio_time yields bogus ~2ms readings.
   • Exact Speech Offset Formula:

     STT_Latency = t_now - (T_stream_start + result_end_offset.total_seconds())

   • datetime.timedelta SDK Invariant: In the Google Cloud Speech v2 Python SDK, result_end_offset is a native datetime.timedelta object (dur.total_seconds()), not a protobuf duration with .nanos.

2. Streaming Live STT (CustomGeminiTranscribeLiveService):

   • Real-Time Turnaround Formula:
     • If user has finished speaking: STT_Latency = t_now - t_user_stopped_speaking (typically 80ms to 350ms).
     • If speech is ongoing in real time: STT_Latency = t_now - t_last_audio_sent (typically 50ms to 150ms).
   • The Utterance Duration Anti-Pattern: Never calculate t_now - t_user_started_speaking as STT latency. That measures the total duration of the spoken sentence (such as 7.8s to 9.1s) rather than speech processing turnaround delay.
   • Turn State Reset Invariant: Immediately upon emitting TranscriptionFrame and calling _handle_transcription(is_final=True), reset self._user_started_speaking_time = None and self._user_stopped_speaking_time = None.

C. Strict Metric Binding, Turn Buffering & Token Event Normalization

• Label Binding: isCascade = this.connectedBotType === "tts-llm-stt".
  • Gemini Live: Displays Live TTFB: {ms}ms.
  • STT-LLM-TTS: Displays STT: {ms}ms | LLM TTFB: {ms}ms | TTS: {ms}ms.
• Async Turn Buffering (pendingLLMLatency): Store early-arriving LLM latencies in a buffer and bind them when creating the bot DOM bubble so they never attach to the previous turn's bubble.
• Cross-Turn Metric Flushing & Isolation:
  • resetMetrics() must purge all latency buffers on every connection start.
  • Clear this.lastTurnSTTLatency = null immediately after rendering on a bubble so previous turn metrics never bleed into subsequent turns.
  • Cap TTFB turnaround timers at < 15.0s to prevent idle timeout skew.
• Token Accumulation & Modality Enum Normalization:
  • Accumulate token and cost metrics strictly once per turn on the discrete usage event (turnComplete), never inside streaming audio chunk callbacks (onMessage) which fire 15 to 30 times per turn.
  • Normalize Python GenAI SDK MediaModality enums (str(m.modality).split(".")[-1].upper()) before JSON serialization so keys like "mediamodality.text" never break modality rate-card lookups in TypeScript.

7. Tool-Calling Architecture, AntiCancel Shield & CallSlots Pattern

A. Session Resumption Handles vs. Stable Session IDs

• Ephemeral Handles: The Live API mints a new session_resumption_handle after every turn.
• Stable Session ID: All turn handles map to a single immutable Session ID (setupComplete.sessionId). Index logs, CRM records, and observability traces by the stable Session ID.

B. Eliminating Redundant Tool-Call Loops

1. Application-Layer State Machines: Move deterministic state routing into the Python or client application layer.
2. Lean System Instructions: Keep persona, voice style, tone, and safety in the root System Instruction; strip out heavy multi-branch decision trees.
3. Dynamic Tool Allow-Lists: Expose only the minimal subset of tools valid for the active conversation state.

C. Non-Blocking Function Calling (behavior: "NON_BLOCKING")

• Setup Declaration: Declare "behavior": "NON_BLOCKING" in function_declarations so the model can speak a natural filler phrase ("Let me check your account details...") while your backend executes the API call.
• Avoid Prompt Hijacking: Never intercept tool calls to dispatch send_client_content("speak exactly...") with turn_complete=True.
• Correct Pattern:
  1. Guide the persona in the System Instruction to acknowledge the lookup naturally before emitting the tool call.
  2. Execute the tool asynchronously in the background via asyncio.create_task(...).
  3. Send tool_response (FunctionResponse) with scheduling="WHEN_IDLE" over the live WebSocket upon completion so the tool result is voiced as soon as the current filler audio finishes without clipping the model's sentence.

D. Tool Response Scheduling Protocol (WHEN_IDLE vs. SILENT vs. INTERRUPT)

• WHEN_IDLE (Default for User-Facing Lookups): Queues the tool result until the model finishes speaking its current filler sentence, then immediately generates the spoken answer.
• SILENT (For Background State Sync Only): Absorbs the tool payload into session context without generating new speech.
  • Caution on Gemini 3.1 Flash Live: If the model speaks a filler sentence ("One moment while I check..."), finishes its turn (turnComplete), and then receives a FunctionResponse with scheduling="SILENT", the model will remain silent until the user speaks again. Always use WHEN_IDLE whenever the caller expects an immediate spoken answer after a filler phrase.

func_response = types.FunctionResponse(
    id=function_id,
    name=function_name,
    response=result_payload,
    scheduling=types.FunctionResponseScheduling.WHEN_IDLE,
)

E. Pipecat AntiCancel Tool Shield, 8.0s Watchdog & Terminal speaking_Hold

When a user coughs or says "okay" while an asynchronous tool call (such as an order lookup or appointment booking) is in flight, Pipecat's default interruption handler cancels the running asyncio task, dropping the API write mid-flight. Protect tool execution with three guards:

1. Override _cancel_function_call(self, function_name: str | None): Log [AntiCancel] Refusing to cancel in-flight function call and return cleanly without cancelling the background asyncio.Task.
2. Frame Interception with 8.0s Watchdog (process_frame): While _tools_in_flight() is active (_active_tools_in_flight > 0 within TOOL_LOCK_MAX_HOLD_SECS = 8.0s), suppress InterruptionFrame, UserStartedSpeakingFrame, and FunctionCallCancelFrame. Never suppress CancelFrame (the pipeline disconnect/shutdown signal, or containers will leak on disconnect).
3. Client-Side speaking_Hold During Terminal Actions (transfer_call / end_call): When the model invokes a terminal tool while speaking a farewell ("Please hold while I connect you to a specialist..."), do not tear down the audio stream while farewell speech is still buffered in the browser. Enter a client-side speaking_Hold state: pause mic forwarding (send_realtime_input), wait for the client to emit playback_drained once the AudioWorklet ring buffer empties, and only then execute the handoff or hangup.

F. The CallSlots Server-State Tool Handler Pattern (Eliminating the 3-Part Tool Token Tax)

On Gemini 3.1 Flash Live, LiveConnectConfig.tools schemas and tool-call responses in session history are re-billed on every subsequent turn. To prevent a multi-tool workflow from adding 2,000+ tokens/turn and 4,800+ tokens of tool-response history bloat:

1. Never Auto-Inject Tool Lists (live_tool_names) into system_instruction:
   • Tool declarations belong exclusively in LiveConnectConfig.tools. Auto-generating an "Available Live API Tools" directory into system_instruction duplicates billing by +650 to 800 tokens/turn. Set live_tool_names=None.
2. Strictly Whitelist FunctionDeclaration Fields (Never Leak HTTP Metadata or Constants):
   • When converting REST tool configs into google.genai.types.FunctionDeclaration objects, strip method, url, headers, and static server fields.
   • Move all static or session-scoped parameters into the server Python handler rather than asking the model to generate constants.
3. Cache Lookup Objects by Monotonic Short IDs (EC-1, EC-2) & Return Compact ~25-Token Summaries:
   • Returning a raw ~800-token JSON array from a lookup tool (such as a branch or clinic finder) leaves those 800 tokens in the Live conversation context for every remaining turn (800 tok * 6 remaining turns = ~4,800 tok/call).
   • Instead, cache the full dictionary objects in server session state (CallSlots.centres) under monotonic query-scoped short IDs (EC-1..EC-3 for pincode 1; EC-4..EC-6 if the caller asks about a second pincode) and return a ~25-token speakable string ("EC-1: Koramangala (2.1 km) | EC-2: Indiranagar (4.0 km)").
   • Always guard lookups with store = self.centres.get(centre_id) so a hallucinated ID (EC-9) returns a speakable recovery prompt rather than raising an unhandled KeyError.
4. Collapse Deterministic Post-Action Tool Chains into the Python Handler:
   • If booking an appointment is always followed by sending a messaging confirmation (232 tok/turn schema) and returning branch contact details (183 tok/turn schema), execute both follow-up API calls directly inside handle_create_appointment_booking() in Python. This removes 2 tool schemas from LiveConnectConfig.tools and collapses 3 sequential LLM tool turns into 1 Python round-trip.

8. Guided State Machine & Phase Engine Architecture

A. Core Architectural Pattern

Structure complex enterprise calls into discrete, predictable Intent Phases:

1. Definite Intent Principle: Define definite_intent (business goal), conversational_boundaries (forbidden actions), and allowed_tools for each phase.
2. Phase-Specific Tool Whitelisting: Restrict execution of action or calculation tools to designated phases. Phases for discovery, consent, and education keep allowed_tools: [].
3. Zero Search Tools for Core Domain Directives: Ground core brand credentials, regulatory disclosures, and safety rules directly in conversational prompt cards with zero tool calls.

@dataclass
class PhaseDefinition:
    phase_id: int
    title: str
    definite_intent: str               # Explicit purpose of this phase
    allowed_tools: List[str]           # Whitelisted tools for this state
    conversational_directive: str      # Spoken behavioral instructions and guidelines
    fast_path_keywords: List[str]      # 0ms regex triggers for immediate state switching

B. Dual-Tier Dynamic Phase Router

+---------------------------------------------------+
|       User Utterance (Live Transcription)         |
+---------------------------------------------------+
                          |
                          v
+---------------------------------------------------+
|         Tier 1: Fast-Path Regex (0ms)             |
+---------------------------------------------------+
          |                               |
  [Match Found]                     [No Regex Match]
          |                               |
          |                               v
          |             +-----------------------------------+
          |             | Tier 2: Async Semantic Classifier |
          |             |     (Flash-Lite, <120ms)          |
          |             +-----------------------------------+
          |                               |
          |                     [Confidence >= 0.70]
          |                               |
          +---------------+---------------+
                          |
                          v
+---------------------------------------------------+
|       await transition_to(target_phase)           |
|      Update Active Phase State & Metrics          |
+---------------------------------------------------+

1. Tier 1 (Instant 0ms Fast-Path): Regex evaluation on transcribed user sentences for high-confidence explicit intents ($0 cost, 0ms latency).
2. Tier 2 (Async Background LLM Classifier): Non-blocking semantic classification using gemini-2.5-flash-lite when Tier 1 finds no match.
3. Hysteresis & Confidence Gate: Only transition states if confidence >= 0.70 and the target phase transition is valid.

C. JIT Prompt Yielding (clientContent) & Mid-Speech Collision Invariant

• Critical Protocol Truth (BidiGenerateContentClientContent Interruption Hazard): Sending BidiGenerateContentClientContent during active model generation unconditionally interrupts the model's in-flight speech (turn_complete=False only avoids starting a new response turn; it does not prevent cutting off active audio generation).
• Mid-Speech Guard & Queue Flushing: Always queue phase transitions that occur while self._is_bot_speaking is true, and flush them inside on_bot_stopped_speaking with turn_complete=False:

async def transition_to(self, target_phase: int, trigger_reason: str):
    async with self._lock:
        if self._is_bot_speaking:
            self._pending_phase = target_phase
            self._pending_reason = trigger_reason
        else:
            await self._yield_prompt_to_gemini(target_phase, trigger_reason)

async def on_bot_stopped_speaking(self):
    self._is_bot_speaking = False
    if self._pending_phase is not None:
        phase, reason = self._pending_phase, self._pending_reason
        self._pending_phase, self._pending_reason = None, None
        await self._yield_prompt_to_gemini(phase, reason)

D. Direct Transcription Hook & Dual-Stream UI Invariant

Override _push_user_transcription inside the LLM service to stream User bubbles to the client UI, record observability traces, and trigger the state machine:

async def _push_user_transcription(self, text: str, result=None):
    await super()._push_user_transcription(text, result)
    if text and text.strip():
        clean_text = text.strip()
        await self.push_frame(OutputTransportMessageFrame(message={
            "label": "rtvi-ai", "type": "server-message",
            "data": {"type": "transcription", "participant": "User", "text": clean_text}
        }))
        GLOBAL_LANGSMITH_TRACER.record_user_turn(clean_text)
        if hasattr(self, "phase_tracker") and self.phase_tracker:
            await self.phase_tracker.handle_user_transcript(clean_text)

E. Deterministic Transition Gates & Consistency Ledger

• Stage Skip Guard: Block anomalous forward jumps greater than N stages (such as max +3) in a single turn.
• Prerequisite Fact Gating: Gated phases cannot be entered until required facts are recorded in the session FactStore.
• Numeric Consistency Ledger: Record all bot-quoted numbers and validate bounds before quoting to guarantee 100% cross-turn consistency.

9. State Machine Disambiguation & Anti-Looping Discipline

A. Disambiguating Turn 1 Greetings vs. Mid-Call Farewells

• The Greeting Loop Anti-Pattern: If callback requests or wrap-up phrases route back to Phase 1, the bot re-introduces itself from the start of the script.
• Strict Funnel Invariant:
  1. Phase 1 (Opening & Availability): Restricted strictly to the Turn 1 opening. Never route back to Phase 1 mid-call.
  2. Phase 9 (Commitment & Close): Any mid-call concluding signal, farewell, or callback request ("bye", "alvida", "thank you bye", "chalo bye", "baad mein baat karte hain") must route to Phase 9.
• Tier-1 Fast-Path Regex for Farewells (0ms, $0):

  if self.current_phase > 1 and re.search(r" (bye|boy|by|alvida|thank you|thanks|chalo bye|ok bye|theek hai bye|wrap up|chalta hu|chalti hu|rakhta hu|rakhti hu|baad mein baat|later) ", lower):
      await self.transition_to(9, trigger_reason="Tier-1 Regex: User wrapping up call / farewell")

B. Dynamic Commitment Validation (No Rigid Scripts)

• In Phase 9, dynamically remind the customer of the specific option explored in the current session (such as the 12-month return plan, minimum deposit, or appointment slot) and confirm whether they are ready to proceed or prefer a scheduled follow-up. Close cleanly with zero loops.

10. Speaker Persona & Grammatical Agreement across Prompt Cards

A. The JIT Prompt Card Persona Drift Trap

When Prompt Cards are injected Just-In-Time via send_client_content(turn_complete=False), the prompt card becomes the most recent instruction in context. If speaker identity and grammatical gender constraints are omitted from the card, multilingual models frequently drift into default masculine Hindi verb forms ("बता रहा हूँ", "देता हूँ").

B. Mandatory Prompt Card Header Standard

Every dynamically yielded Prompt Card must include the speaker identity and grammatical agreement rules using natural-language prefixes (never square brackets, which native audio models can vocalize aloud):

directive_text = (
    f"System update for agent - Active Phase {target_phase} ({card['title']}):\n"
    f"Speaker Persona: Pragya (Female Senior Wealth Manager at Cymbal Lending).\n"
    f"Mandatory Female Grammar: Always speak in 100% consistent feminine Hindi grammar for yourself.\n"
    f"• REQUIRED FEMININE VERBS: 'मैं बता रही हूँ', 'करती हूँ', 'देती हूँ', 'मदद करूँगी', 'समझ गई'\n"
    f"• FORBIDDEN MASCULINE VERBS: Never use masculine verb forms ('रहा हूँ', 'करता हूँ', 'देता हूँ', 'करूँगा').\n\n"
    f"{card['directive']}\n\n"
    f"Context: {trigger_reason}\n"
    f"Rule: Always use Devanagari for Hindi words and Latin script for English financial terms."
)

11. Sub-Millisecond Near-Process Dual-Layer RAG Caching

A. The Remote Vector Search Dead Air Hazard

• The Anti-Pattern: Calling remote vector database search APIs inside an active voice turn takes 3,200ms to 3,800ms, introducing over 3 seconds of dead air.
• Dual-Layer Near-Process Hierarchy:
  1. L1 In-Memory Process RAM (<0.05ms): Compile domain Q&A pairs into an Okapi BM25 inverted index in process memory on startup for sub-millisecond retrieval.
  2. L2 Cloud Memorystore Valkey / Redis (~1.0ms): Use a regional GCP Memorystore instance in us-central1 for cross-instance shared state and normalized query hash keys (cymbal:sheet_rag:*).
  3. Graceful Fallback: If L2 Redis is unreachable, fall back immediately to the local L1 RAM BM25 index without blocking the live voice stream.
• Proactive vs. Reactive RAG Injection:
  • Proactive Background RAG (Between Turns): Inject retrieved context silently via LiveClientContent(turns=[Content(role="user", parts=[Part.from_text("System context update: ...")])], turn_complete=False). Because turn_complete=False, the model absorbs the context without generating unsolicited speech.
  • Reactive On-Demand RAG (User Asks a Specific Question): Expose a non-blocking search_knowledge_base tool returning FunctionResponse(..., scheduling="WHEN_IDLE").

12. Dual-Brain Frontcar / Downcar Architecture & Enterprise Memory Bank

A. Frontcar vs. Downcar Decoupling

• Frontcar (Fast Path - Synchronous Duplex Voice): Ultra-lean real-time pipeline (gemini-3.1-flash-live-preview or gemini-live-2.5-flash-native-audio) focused on conversational persona, audio pacing, active phase intent, and non-blocking tool execution. At session connect (t = 0ms), inject the user's compact Markdown profile (< 500 tokens) into system_instruction. Zero heavy database writes occur on the audio thread.
• Downcar (Slow Path - Asynchronous Post-Session Worker): Triggered on on_client_disconnected. Uses gemini-2.5-flash-lite (response_mime_type="application/json") to parse session transcripts, extract canonical user facts and action items, generate episodic summaries, and persist them to your cloud memory store.

B. Dual-Threshold Vector Memory Standard

• Deduplication Gate (SIMILARITY_THRESHOLD = 0.83): Enforce cosine_similarity >= 0.83 (plus category match and MD5 content_hash) when updating or inserting memory facts so distinct concepts never overwrite each other.
• Retrieval Gate (RETRIEVAL_THRESHOLD = 0.40): For semantic recall (retrieve_memory), use threshold = 0.40. Conversational questions embed at 0.65 to 0.78 cosine similarity against declarative facts; using 0.83 for retrieval causes false-negative recall misses.
• Stateless Container Rule: Never store user facts in local container files (/tmp or local SQLite) on Cloud Run. Persist to a managed cloud store (Firestore, Cloud SQL, or Vertex AI Agent Engine Memory Bank).

C. Spoken Lexical User Identity Normalization

Standardize spoken introductions by stripping conversational framing ("My name is..."), removing boundary-aware honorifics (Dr., Mr., Smt.), and sanitizing symbols while preserving alphanumeric IDs (user_<name>).

13. Token Modality, Cost Accounting & Context Compression Architecture

A. Published Rate Card (Vertex AI & Google AI Studio)

B. Dual-Engine Optimization Architecture: 5,000-Token Trigger Floor vs. Lean SI & JIT Prompt Cards

• The 5,000-Token Trigger Invariant: Vertex AI Live enforces a minimum threshold of 5,000 tokens (trigger_tokens=5000, target_tokens=3500) for sliding-window context compression.
• The Lean SI Mismatch:
  • In an optimized voice pipeline, the root System Instruction (SI) is intentionally compact (~225 to 500 tokens) containing only core persona, tone, grammatical rules, and safety boundaries.
  • Because the Live protocol is append-only, every injected prompt card and every turn of user and bot speech accumulates in session history until the 5,000-token trigger is reached (typically around Turn 7 to 12).
• The Dual-Engine Solution:
  1. Early Turns (Lean Static SI + Dynamic JIT Prompt Cards):
     • Keep the root SI under 500 tokens (120 tokens for the opening card, ~500 tokens average across phases).
     • Yield Phase Directives and compliance scripts Just-In-Time via send_client_content(turn_complete=False) only when entering a new conversation phase.
     • Never stuff static multi-page decision trees or unused tool schemas into the root SI.
  2. Long Calls (Sliding-Window Compression 5000 / 3500 + FactStore):
     • Once total context reaches 5,000 tokens, SlidingWindow(target_tokens=3500) automatically evicts the oldest dialogue turns (including their accumulated user and model audio tokens) while preserving the initial system_instruction.
     • FactStore (< 60 tokens) restores structured continuity across compaction boundaries.

C. Bidirectional Audio Accumulation & Sliding-Window Eviction Dynamics

• Why Prompt AUDIO Grows Rapidly (UserInputAudio + PlayedModelOutputAudio):
  • Unlike text-only APIs where prior assistant replies are compact text tokens, a native Speech-to-Speech session retains both the user's input speech audio and the model's played output audio in the active conversation history (Prompt AUDIO, billed at $3.00 / 1M on every subsequent turn), alongside their text transcripts in Prompt TEXT:

    Prompt_AUDIO(turn k) = sum_{i=1..k} UserInputAudio(i) + sum_{i=1..k-1} PlayedModelOutputAudio(i)

  • Production Telemetry Proof (Vertex AI gemini-live-2.5-flash-native-audio, 3 Tools Declared):
    • Turn 1: Prompt = 279 (236 TEXT = 211 SI + 25 user ASR text, 43 AUDIO user greeting), Response = 223 (181 AUDIO played bot speech, 42 TEXT bot transcript).
    • Turn 2: Prompt = 556 (AUDIO = 271 = 43 Turn 1 user audio + 181 Turn 1 played bot audio + 47 Turn 2 user audio; TEXT = 285 = 236 Turn 1 text + 42 Turn 1 bot text + 7 Turn 2 user ASR text), Response = 212 (170 AUDIO, 42 TEXT).
• Barge-In Truncation Reduces Subsequent Prompt AUDIO:
  • When a caller interrupts the model mid-sentence (barge-in), the server truncates the unplayed tail of the model's audio turn from session history based on playback duration. On the next turn, only the model audio tokens actually played before the interruption are carried forward into Prompt AUDIO.
  • Automated Benchmark Trap: Because the server finishes streaming audio packets over the WebSocket faster than real-time playback duration (generated_audio_duration), an automated test script that sends client_content(turn_complete=True) immediately upon receiving turnComplete without waiting for real-time audio playout triggers server-side barge-in truncation, slicing the model's audio out of history and under-measuring real production Prompt AUDIO growth.
• Sliding-Window Audio Eviction:
  • When ContextWindowCompressionConfig(trigger_tokens=5000, sliding_window=SlidingWindow(target_tokens=3500)) fires, the server evicts the oldest conversation turns (for example contracting Prompt AUDIO from 2,689 -> 1,794 and 2,015 -> 799 tokens) while keeping the root system_instruction intact.
  • Because audio input costs $3.00 / 1M vs. $0.50 to $0.75 / 1M for text input (4x to 6x rate ratio), evicting 1,216 audio tokens saves the cost equivalent of 4,864 to 7,296 text tokens.

D. FactStore Restoration Protocol & 10-Turn Cap Invariant

• The Context Amnesia Footgun: When sliding-window compaction evicts early dialogue turns, the model can forget facts stated in Turns 1 to 3 (such as customer name, pin code, or loan amount) unless those facts are preserved cleanly.
• The FactStore Invariant:
  • Detect compaction via current_prompt < (last_prompt - 150).
  • Maintain a compact < 60-token JSON FactStore of extracted entities plus a capped window of recent transcript lines, and inject them via send_client_content(turns=[Content(...)], turn_complete=False).
• The 10-Turn Cap Rule:
  • Never re-inject the entire session transcript on a 30-turn call (which would inject 1,250+ text tokens and defeat compression).
  • Strictly cap injected dialogue to the last 10 turns (history[-10:]):

    capped_history = history[-10:] if len(history) > 10 else history
    injected_lines = [f"{turn['role']}: {turn['text']}" for turn in capped_history]

• Prompt Restoration Template (Zero Bracket Tags):

  System update for agent - Conversation Transcript Log (Last 10 Turns):
  <injected_turns>
  
  Instructions for agent:
  - Continue the conversation with the caller from the latest turn.
  - Retain all customer details, numbers, preferences, and agreements stated above.
  - Do not repeat greetings, do not re-introduce yourself, and do not verbally acknowledge this transcript update.

E. Total Transcription History Backend Observability Invariant

• Zero Silent Drops: Log every dialogue turn in real time and in session summaries within the backend logger:
  1. Per-Turn Logs: Log [Transcript User (Turn N)]: <text> and [Transcript Assistant (Turn N)]: <text>.
  2. At Compaction Injection: Log the complete session transcription history alongside the 10-turn restoration payload.
  3. At Client Disconnect: In on_client_disconnected, emit the numbered session transcript (1. User: ..., 2. Assistant: ...).

F. Audio vs. Text Token Dynamics & Compression Alert Guarding

• Composite Prompt Tokens: On Vertex AI, prompt_token_count equals Prompt_TEXT + Prompt_AUDIO (residual = 0). On Google AI Studio (gemini-3.1-flash-live-preview), prompt_token_count exceeds Prompt_TEXT + Prompt_AUDIO by 93 to 221 tokens per turn (see §13.H). Never compute billed volume by summing prompt_tokens_details alone; always read prompt_token_count and bill any positive difference max(0, prompt_token_count - sum(details)) at the text input rate.
• Guard Observability Alerts: Configure monitoring dashboards so context-compression alerts only fire when prompt_token_count crosses >= 4,500 tokens followed by a sustained drop (> 1,000 tokens).

G. Tool Declaration & Tool Response Token Accounting (Prompt TEXT)

• Gemini 2.5 Flash Native Audio (0 Tool Schema Tokens in prompt_token_count): On gemini-live-2.5-flash-native-audio, the turn-by-turn prompt_token_count counts system_instruction text tokens (225 tok) and omits declared tools (function_declarations) schemas (0 billed tool schema tokens per turn), billing tool payloads only when tools are invoked and their FunctionResponse payloads enter conversation history.
• Gemini 3.x Live Architecture (Full Developer-Turn Tool Schema Billing + History Carry): On 3.x Live pipelines where function_declarations are serialized into the developer system turn (as well as every returned FunctionResponse in session history), declared tool schemas (~70 to 250 tokens per concise tool, or 2,300 to 3,500+ tokens/turn for 16 verbose tools) and tool outputs are re-billed on every subsequent turn.
• Pruning Invariant: Register only the 1 to 3 essential real-time tools required for the active workflow and apply the CallSlots pattern (§7.F) so neither tool schemas nor 800-token JSON lookup responses bloat your 5,000-token context window.

H. Cross-Endpoint usage_metadata Reconciliation Invariant (Vertex AI vs. AI Studio)

Paired multi-turn sessions on identical Pipecat clients and prompts establish exact reconciliation behaviors across endpoints:

1. Google AI Studio (gemini-3.1-flash-live-preview) Unattributed Prompt Residual:

   T0 (greeting): Prompt: 795  (TEXT 501, AUDIO 201)  -> details sum 702   residual  93
   T1           : Prompt: 1091 (TEXT 538, AUDIO 418)  -> details sum 956   residual 135
   T2           : Prompt: 1343 (TEXT 578, AUDIO 594)  -> details sum 1172  residual 171
   T3           : Prompt: 1692 (TEXT 633, AUDIO 838)  -> details sum 1471  residual 221
                                                         TOTAL RESIDUAL    620 = 12.6%

   • On AI Studio, response_tokens_details reports AUDIO only (0 TEXT out), while prompt_token_count includes internal turn formatting and the carried output text transcript that are not broken out in prompt_tokens_details.
   • AI Studio also sets total_token_count = prompt_token_count + response_token_count, omitting thoughts_token_count from total_token_count even when thoughts_token_count is populated.
2. Vertex AI (gemini-live-2.5-flash-native-audio) Zero Residual:
   • On Vertex AI, prompt_token_count == Prompt_TEXT + Prompt_AUDIO (residual = 0 on every turn), response_tokens_details explicitly breaks out both AUDIO and TEXT, and total_token_count == prompt_token_count + response_token_count + thoughts_token_count.
3. Cost-Calculator Invariant:
   • Any Live pricing module must compute residual_input = max(0, prompt_token_count - sum(prompt_tokens_details)) and include thoughts_token_count explicitly when computing total turn cost.

I. Pre-Speech Greeting Audio Metering on AI Studio (gemini-3.1-flash-live-preview)

On Google AI Studio, gemini-3.1-flash-live-preview bills ~196 to 201 audio prompt tokens on Turn 1 before the user has spoken if the client microphone stream is already open during the bot's greeting playback:

05:58:48.8  WebSocket connected (mic streaming silence)
05:58:49.8  Bot started speaking (greeting)
05:58:56.4  usage_metadata: Prompt: 795 (TEXT: 501, AUDIO: 201)   # 7.6s of open mic

Because the Live API re-bills active context on every subsequent turn, those 201 Turn-1 silence tokens compound to 804+ extra audio tokens over 4 turns.

• Mitigation: Gate client microphone forwarding until TTSStoppedFrame (or client greeting playback drain) fires on Turn 1. When Turn 1 is triggered cleanly without open-mic audio before the greeting completes, Turn 1 Prompt AUDIO is 0.

J. Zero Implicit Context Caching Discount (cached_content_token_count = 0)

Across all production Gemini Live endpoints (gemini-live-2.5-flash-native-audio on Vertex AI and gemini-3.1-flash-live-preview on Google AI Studio), cached_content_token_count is 0 (None) on every turn. System instructions, declared tool schemas (on 3.1 Live), and carried conversation history in the active window are billed at standard input rates on every turn:

What can look at first glance like "2.5 caching the system prompt" is actually 2.5-flash-native-audio having a smaller fixed preamble (225 vs. 501 tokens) and omitting tool schemas from prompt_token_count. Both models re-bill their active context window on every turn (~2.9x cumulative re-bill amplification over 4 to 5 turns).

K. Thinking Token Accounting (thoughts_token_count)

• On gemini-live-2.5-flash-native-audio with thinking_budget=0 and gemini-3.1-flash-live-preview with ThinkingLevel.MINIMAL (include_thoughts=False), thoughts_token_count is 0 (None) on standard conversational turns.
• Always keep thinking_level="MINIMAL" and include_thoughts=False on 3.1 Live for real-time voice agents; higher thinking levels (LOW, MEDIUM, or HIGH) add both turn latency and output-rate thinking token spend ($4.50 / 1M) across multi-turn calls.

L. Parametric Per-Call Cost Model & Turn-by-Turn Calculator

Because both user speech (u_aud) and played model output audio (b_aud_kept) accumulate in Prompt AUDIO on every turn until SlidingWindow(5000/3500) triggers, and both user ASR text (u_txt) and model output text (b_txt) accumulate in Prompt TEXT:

def calculate_live_session_cost(
    si_tokens: int = 500,
    tool_tokens: int = 200,
    turns: int = 6,
    user_sec_per_turn: float = 4.0,
    bot_sec_per_turn: float = 6.0,
    barge_in_play_ratio: float = 1.0,
    model: str = "gemini-3.1-flash-live-preview",
    sliding_window_trigger: int = 5000,
    sliding_window_target: int = 3500,
) -> dict:
    # Turn-by-turn token and USD calculator matching Gemini Live production billing.
    u_aud = int(round(user_sec_per_turn * 25))
    b_aud_gen = int(round(bot_sec_per_turn * 25))
    b_aud_kept = int(round(b_aud_gen * barge_in_play_ratio))
    u_txt = int(round(u_aud * 0.15))
    b_txt = int(round(b_aud_gen * 0.25))

    if model == "gemini-live-2.5-flash-native-audio":
        r_txt_in, r_txt_out, r_aud_in, r_aud_out = 0.50, 2.00, 3.00, 12.00
        base_txt = si_tokens  # 2.5 omits static tool_tokens from prompt_token_count
        thoughts_per_turn = 0
    else:
        r_txt_in, r_txt_out, r_aud_in, r_aud_out = 0.75, 4.50, 3.00, 12.00
        base_txt = si_tokens + 276 + tool_tokens  # 3.1 adds +276 preamble scaffolding
        thoughts_per_turn = 0  # 0 with ThinkingLevel.MINIMAL

    cum_txt_in = 0
    cum_aud_in = 0
    hist_turns = []  # list of (turn_txt, turn_aud)

    for _ in range(1, turns + 1):
        hist_txt = sum(x[0] for x in hist_turns)
        hist_aud = sum(x[1] for x in hist_turns)
        cur_txt = base_txt + hist_txt + u_txt
        cur_aud = hist_aud + u_aud

        if sliding_window_trigger and (cur_txt + cur_aud) > sliding_window_trigger:
            while hist_turns and ( base_txt + sum(x[0] for x in hist_turns) + u_txt + sum(x[1] for x in hist_turns) + u_aud ) > sliding_window_target:
                hist_turns.pop(0)
            cur_txt = base_txt + sum(x[0] for x in hist_turns) + u_txt
            cur_aud = sum(x[1] for x in hist_turns) + u_aud

        cum_txt_in += cur_txt
        cum_aud_in += cur_aud
        hist_turns.append((u_txt + b_txt, u_aud + b_aud_kept))

    resp_aud = turns * b_aud_gen
    resp_txt = turns * (b_txt + thoughts_per_turn)

    usd = (
        cum_txt_in * r_txt_in
        + cum_aud_in * r_aud_in
        + resp_aud * r_aud_out
        + resp_txt * r_txt_out
    ) / 1e6

    return {
        "prompt_text_tokens": cum_txt_in,
        "prompt_audio_tokens": cum_aud_in,
        "response_audio_tokens": resp_aud,
        "response_text_and_thought_tokens": resp_txt,
        "total_usd": round(usd, 5),
    }

M. Duplex Voice Compounding Economics: The "Carried Audio Tax" & Who Drives Call Cost

Empirical measurements on an 18-turn uncompressed session (54,315 total billed tokens, $0.1178 on Gemini 2.5 Flash Native Audio) illustrate why carried audio history dominates multi-turn voice spend:

+------------------------------------------------------------------------+
|  Audio Prompt Input (Carried Audio History) : 25,365 tok -> $0.0761 (64.6%) |
|  Audio Response Output (Generated Speech)   :  2,268 tok -> $0.0272 (23.1%) |
|  Text Prompt Input (System & Cards)         : 25,904 tok -> $0.0130 (11.0%) |
|  Text Response Output (Transcripts)         :    778 tok -> $0.0016  (1.3%) |
|  --------------------------------------------------------------------  |
|  TOTAL                                      : 54,315 tok -> $0.1178 (100%)  |
+------------------------------------------------------------------------+

1. Why $12.00 / 1M Audio Output Is Only 23.1% of Total Spend:
   • Although generated bot speech has the highest unit price ($12.00 / 1M tokens = $0.018 / min), Candidate AUDIO is billed only once on the turn it is spoken.
2. Why Prompt AUDIO ($3.00 / 1M) Drives 64.6% of the Call Bill:
   • Every turn's user audio and played bot audio remain in Prompt AUDIO and are re-billed at $3.00 / 1M on every subsequent turn until SlidingWindow(5000/3500) evicts old turns.
   • By Turn 14 of an uncompressed call (2,363 accumulated audio tokens in prompt), even a one-word user acknowledgment ("Yes") re-processes all 2,363 prior audio tokens at $3.00 / 1M.
   • Note why the 18-turn trace above cost $0.1178: it already used a compact ~500-token root prompt (25,904 cumulative text tokens across 18 turns) rather than a monolithic 4,500-token static SOP (81,000 cumulative text tokens).
   • Combining all three optimization levers (Pillar 1 SlidingWindow(5000/3500) for long calls + Pillar 2 JIT Prompt Cards capping active instructions at ~500 tokens + Pillar 3 rolling audio-to-text pruning / FactStore < 60 tokens for calls under 15 turns) reduces a 3-minute, 18-turn enterprise call from $0.246 ($0.082/min with a 4,500-token static prompt and unpruned audio) down to $0.093 ($0.031/min, a 62% reduction).

N. Interruption Economics & Barge-In Math (Output Savings vs. Extra-Turn Prompt Fee)

When a user interrupts the bot mid-speech (server_content.interrupted == True), two opposing cost effects occur:

1. What You Save (Aborted Output Audio + Truncated History):
   • The server stops generating remaining audio (saving $12.00 / 1M on ungenerated output tokens) and truncates the unplayed tail of the bot's turn so unplayed audio tokens are not carried forward into future Prompt AUDIO turns.
   • Example: Cutting off a 300-token bot monologue after 50 played tokens saves 250 output audio tokens ($0.0030) plus 250 * $3.00 / 1M = $0.00075 on every subsequent turn.
2. The Hidden Trap (The "Micro-Interruption Prompt Fee"):
   • Input prompt evaluation for the interrupted turn has already occurred and been billed before the first audio chunk was generated.
   • If the interruption was a false barge-in (a cough, throat clear, or backchannel "uh-huh" that triggers an extra conversational turn), the model must run a brand-new prompt evaluation across the entire active context window (3,500 to 4,000 tokens):

     Extra Turn Prompt Cost = (2,300 audio tok * $3.00/1M) + (1,700 text tok * $0.50/1M) = $0.00775
     Net Impact of False Barge-In = $0.00300 saved - $0.00775 extra turn = -$0.00475 net loss

3. Golden Rule of Interruption:
   • Decisive Interruption (Saves Money): Caller cuts off an unwanted explanation and pivots directly to the next step ("Skip that, book the 10 AM slot"), replacing a turn and truncating carried bot audio.
   • Fragmented False Barge-In (Burns Money): Coughs or backchannel fillers ("uh-huh") split one turn into two, re-billing the entire context history. Tune StartSensitivity.START_SENSITIVITY_LOW and SileroVADAnalyzer(confidence=0.7, start_secs=0.2) to filter out non-speech bursts.

O. Model Comparison & Production Decision Matrix (2.5 Native Audio vs. 3.1 Flash Live)

P. Enterprise Voice Agent Design Principles

1. Target 6 to 8 Turns for Transactional Funnels:
   • A focused 6-turn booking or verification call stays well below the 5,000-token compaction trigger and costs ~$0.025 to $0.035 in total.
2. Enable SlidingWindow(5000, 3500) on Every Session:
   • Keeps active context between 3,500 and 5,000 tokens so 15-to-30-minute support calls scale linearly (~$0.06/min with SlidingWindow alone, or ~$0.031/min when combined with JIT prompt cards and rolling audio-to-text pruning as shown in §13.M) rather than quadratically.
3. Decouple UI Progress Telemetry from Model Tool Calling:
   • Never rely on the voice model to call a get_phase_card or track_progress tool just to update the browser UI.
   • Run server-side regex and async Flash-Lite transcription tracking (PhaseTracker) on incoming user transcripts for 0ms voice latency, $0 Live tool token overhead, and deterministic UI state updates.
4. Gate the Microphone on Turn 1:
   • Keep client mic audio gated until the opening greeting finishes playing (TTSStoppedFrame) so ambient room noise during the greeting is never metered into Turn 1 Prompt AUDIO.

Q. The Turn-0 Cold-Start Trap: Why Initial Tokens Ballooned to 1,254

Truncated - read the full file at https://github.com/manishkjs/gemini_live_pipecat/blob/83ae466442067cf0d8a5845c1c4462115353fd6d/docs-site/public/gemini-live/SKILL.md.