covenant_sentinel

Imported from el-noir/covenant_sentinel (AGENTS.md). Install upstream with npx skills add el-noir/covenant_sentinel. Copyright stays with the author.

Covenant Sentinel

A financial covenant-monitoring agent built for the Vultr Cloud x 24 Raise Summit (Statement Two track). It accepts real credit agreement PDFs and pre-loaded SEC documents, extracts covenant clauses via LLM, calculates financial ratios via genuine LLM tool-calling on Vultr kimi-k2.6, checks historical trends, makes a preliminary decision, then challenges and reconciles its own findings before producing a confidence-scored memo.

Pitch (One-Liner)

"Covenant Sentinel reads your actual credit agreement, extracts the covenants, and argues with itself before escalating. Finance teams stop drowning in false-positive alerts."

Hackathon Constraints (HARD RULES)

Do NOT build any of these — they result in immediate disqualification:

• Streamlit applications — If we ship Streamlit, we are disqualified.
• Basic RAG applications — A single retrieve-then-answer pipeline is banned.
• Dashboard as the main feature — Passive data viewers are banned.
• Presentations, not technical demos — Judges want screen recordings of actual software, not slides.

What This Means for Architecture

1. Next.js is mandatory, not optional. We ship a real web app with a build step.
2. The frontend must show the agent reasoning, not just the result. The StepTrace component that visualizes plan → retrieve → calc → decide → challenge → reconcile → memo is not a decoration — it is the primary feature. It proves this is an agent, not a dashboard.
3. The adversarial layer is non-negotiable. Without challenge → reconcile, this is basic RAG + a flag. The self-checking loop is what transforms a retrieval pipeline into an agent.
4. Demo video = 50% of the score. Every architectural decision must optimize for what is visible in a 60-second screen recording.

Data Input Model — Dual Mode

The agent supports two ways to provide documents:

1. Pre-loaded Collection (Default for demo): User selects a real company (MosaicCo, CovestroAG). Data comes from data/covenants.json — real SEC credit agreement covenant sections with representative legal language.
2. PDF Upload (Feature): User uploads any credit agreement PDF. The backend parses text with PyPDF2, uses an LLM to extract covenant clauses / thresholds / formulas, validates the extraction, stores it temporarily, then runs the full analysis.

Demo strategy: The 60-second video only uses pre-loaded mode to guarantee stability. The upload tab is visible in the UI as proof of capability but is not triggered live unless a judge asks during Q&A.

Architecture

State Schema (TypedDict)

class AgentState(TypedDict):
    company_id: str
    covenant_type: str          # "fixed_charge_coverage" | "leverage_ratio"
    plan: str                   # LLM reasoning for selected covenant
    clause_text: str            # Full clause text retrieved
    clause_citation: str        # e.g. "Section 6.1, Credit Agreement dated March 14, 2024"
    financial_inputs: dict      # {"ebitda": 1200000, "fixed_charges": 810000}
    trend_values: list[float]   # Last 4 quarters
    ratio_value: float
    threshold: float
    ratio_formula: str
    preliminary_decision: str   # "BREACH" | "NO_BREACH" | "WATCH"
    preliminary_confidence: float
    challenge_arguments: str    # Devil's advocate output
    final_decision: str
    final_confidence: float
    memo: str
    _loop_count: int            # Guards adversarial cycles (max 1)

Graph Flow

The agent is a ReAct-style multi-step loop. At key decision points (calc_agent and memo_agent), the LLM is bound to a shared tool pool. If it decides a tool is needed, execution routes to a single ToolNode, which executes the chosen tool and returns the result.

plan -> retrieve_clause -> calc_agent [bind_tools]
  -> [conditional: tool_calls?] -> ToolNode -> calc_agent ...
  -> retrieve_trend -> decide [conditional]
    if BREACH:
      -> challenge -> reconcile [conditional]
        if confidence < 0.6 and _loop_count < 1:
          -> retrieve_clause (loop back)
        else:
          -> memo_agent [bind_tools] -> ToolNode -> memo_agent -> END
    else:
      -> memo_agent [bind_tools] -> ToolNode -> memo_agent -> END

ReAct Tool Loop

• calc_agent and memo_agent are LLM nodes that see the full conversation history + available tools (CalculateFinancialRatio, DraftMemo).
• If the LLM returns tool_calls, the graph routes to ToolNode, which invokes the correct tool implementation from tools.py.
• ToolNode output is appended to the message history and fed back to the same LLM node.
• The loop ends when the LLM returns no more tool_calls, and execution proceeds to the next business-logic node.
• This pattern is not two separate one-off tool calls — it is genuine multi-turn reasoning with tool use at multiple points in the graph.

Conventions

• backend/nodes.py — All node implementations. Pure functions returning partial state updates.
• backend/prompts.py — All system prompts. Prompts are constants or factory functions, never inline strings in nodes.py.
• backend/graph.py — StateGraph builder with conditional edge routers.
• backend/tools.py — Pydantic tool schemas + deterministic fallback implementations.
• backend/state.py — TypedDict schema.
• backend/main.py — FastAPI app. Endpoints: POST /upload (PDF) + POST /run (analysis).
• backend/pdf_parser.py — PyPDF2 text extraction + LLM covenant extraction with validation.
• backend/data/store.py — Document store. Reads from covenants.json for pre-loaded docs; stores uploaded docs in memory/JSON.
• backend/data/covenants.json — Pre-fetched real SEC credit agreement covenant sections.
• frontend/app/page.tsx — Single Next.js page. Two input tabs: "Select Company" and "Upload Document".
• frontend/components/StepTrace.tsx — PRIMARY FEATURE. Visual graph trace showing every executed node in real-time.
• frontend/components/MemoDisplay.tsx — Formatted memo with decision badge and confidence bar.
• frontend/components/AnalysisForm.tsx — Dual-mode input form (dropdown + file upload).

API Contracts

POST /upload

Content-Type: multipart/form-data
file: <raw.pdf>

Response: { "company_id": str, "covenants": dict } — extracted data. On failure: HTTP 400.

POST /run

{"company_id": "MosaicCo", "covenant_type": "fixed_charge_coverage"}

Response: Full AgentState JSON. Frontend renders final_decision, final_confidence, memo, clause_citation, and challenge_arguments.

Tool Calling Requirement

The agent runs a ReAct-style tool loop using a shared ToolNode:

from langgraph.prebuilt import ToolNode

tools = [CalculateFinancialRatio, DraftMemo]
tool_node = ToolNode(tools)

Bound tools:

• CalculateFinancialRatio — computes ratio from financial inputs
• DraftMemo — generates the final structured memo with citations

How it works:

• calc_agent and memo_agent are LLM nodes with vultr_llm.bind_tools(tools).
• If the LLM emits tool_calls, execution routes to ToolNode, which executes the tool and returns the result.
• The result is appended to the message history and fed back to the LLM node.
• When no more tool_calls are present, the graph proceeds to the next node.

Rubric value: This demonstrates multiple distinct tool decisions (calculation + document generation) in a single run. The LLM actively chooses which tool to call and when. If the API returns no tool_calls, deterministic fallbacks in tools.py still return correct values, but a warning is logged.

Adversarial Layer Rules

• challenge node ALWAYS activates on preliminary_decision == "BREACH".
• reconcile produces final_decision and final_confidence.
• If final_confidence < 0.6 and _loop_count < 1, route back to retrieve_clause once, then forced memo.
• Challenge prompt must reference specific clause text (e.g., "Section 6.1 excludes non-recurring items").
• Generic challenges are disqualifying. The counter-argument must cite the exact clause and propose a specific adjustment.

Data Strategy

• Pre-fetch real SEC credit agreement exhibits into covenants.json.
• The data/store.py module abstracts retrieval so the graph node always says "retrieving from document store" regardless of whether it's a pre-loaded JSON entry or an uploaded PDF extraction.
• Uploaded PDFs are parsed, extracted by LLM, validated (threshold must be numeric > 0, formula must contain expected keywords), then stored temporarily.

Key Files

Environment

• VULTR_SERVERLESS_INFERENCE_API_KEY — Required. From console.vultr.com > Serverless > Inference.
• LANGCHAIN_API_KEY — Optional but recommended. Enables LangSmith tracing of graph executions. Invaluable for demonstrating multi-step reasoning to judges.
• GRADIUM_API_KEY — Optional. For TTS "Play Memo" feature. Get 45,000 credits at gradium.ai + 100,000 with coupon RAISE-2026.
• Python 3.10+ (for | union types, TypedDict, etc.)
• Node.js 20+ (for Next.js frontend)

Judging Criteria Mapping

60-Second Demo Video Script

0:00–0:08: User opens app. Selects "Select Company" tab. Chooses "MosaicCo", "Fixed Charge Coverage". Clicks Run Analysis. Animated StepTrace: plan → retrieve_clause → calc_ratio → retrieve_trend → decide.

0:08–0:18: "Calculated Ratio: 1.48" (below 1.50). StepTrace decide in red. "Preliminary: BREACH (confidence: 0.72)".

0:18–0:35: Money shot. StepTrace → challenge. Panel: "Devil's Advocate Report" — "Section 6.1 excludes non-recurring items. The $50K restructuring charge in Q2 is likely one-off and should be excluded."

0:35–0:48: StepTrace → reconcile. "Final Decision: WATCH (confidence: 0.68)". Confidence bar drops.

0:48–0:58: Final memo with inline citation to Section 6.1.

0:58–1:00: Close card: "Covenant Sentinel — built on LangGraph + Vultr Serverless Inference in 24 hours."

Demo Scenario (Test Case)

• Company: MosaicCo
• Covenant: fixed_charge_coverage
• Inputs: EBITDA = 1.2M, Fixed Charges = 810K
• Ratio: 1.48
• Threshold: 1.50
• Preliminary: BREACH (confidence 0.72)
• Challenge: cites Section 6.1 exclusion, questions Q2 restructuring charge
• Reconcile: shifts to WATCH, confidence drops to 0.68
• Final memo: cites both raw calculation and challenge reasoning

Failure Modes (Handled)

1. Tool call fails / no tool_calls → Deterministic fallback, log warning.
2. Missing company/covenant → FastAPI 400.
3. LLM refuses structured output → Set confidence to 0.5, route to memo.
4. Adversarial infinite loop → _loop_count hard-cap at 1.
5. Vultr API rate limit → Exponential backoff (1s, 2s, 4s), then 503.
6. PDF upload fails / unreadable → /upload returns 400: "Could not extract text from PDF."
7. LLM hallucinates threshold / formula → Validation layer rejects non-numeric thresholds or formulas missing "EBITDA"/"Debt".

Future Tool Pool (Post-Hackathon)

The current tool pool is intentionally scoped for 24-hour stability:

• CalculateFinancialRatio — computation
• DraftMemo — document generation

Potential additions (not implemented for hackathon):

Rationale for deferral: The rubric evaluates "calls tools" — not "calls many tools." Two genuine tool decisions (calculation + memo) satisfy the requirement. External tools add infrastructure overhead with minimal incremental judging value in a 60-second demo. They are documented here for post-hackathon expansion.

Code Style

• Python: black, ruff, strict mypy typing. No bare dict where TypedDict or Pydantic applies.
• TypeScript: strict mode, functional components, no any.
• Never commit API keys. .env is in .gitignore.