Azureus-FinTech

Imported from YelamanKarassay/Azureus-FinTech (AGENTS.md). Install upstream with npx skills add YelamanKarassay/Azureus-FinTech. Copyright stays with the author.

AGENTS.md — Azureus Project Conventions for Codex

This file is read by Codex at the start of every session. Keep it short, opinionated, and current. When project conventions change, update this file first.

What This Project Is

Azureus is an open-source educational research platform for systematic equity strategies on Hong Kong equities. Backtester + showcase strategies + interactive visualization. Deployed at azureus.tech. MIT licensed.

For full context, read docs/ARCHITECTURE.md. You should reference it before making any structural decision.

How to Work in This Repo

Read these files first, in this order:

1. This file (AGENTS.md) — conventions and rules
2. docs/ARCHITECTURE.md — full architectural decisions and rationale
3. The relevant section's locked block in ARCHITECTURE.md before changing that section's code
4. The current phase's success criteria in ARCHITECTURE.md Section 9

When in doubt, ask. When not in doubt, also consider asking.

This project is being built with deliberate trade-offs. Defaults from web tutorials are often wrong for this codebase. If you're tempted to suggest something not in ARCHITECTURE.md, treat that as a signal to confirm with the developer before writing code.

Stack (Locked — Do Not Substitute)

If a task requires a tool not on this list, stop and ask before installing it.

Hard Rules — Violating These Breaks the Project

Rule 1: PIT (Point-In-Time) Correctness Is Non-Negotiable

Every fundamental record stores both period_end (fiscal date) and reported_date (announcement date). Every macro record stores both date (value-date) and reported_date. Strategy code never queries raw tables — only through DataSource methods that take as_of_date and filter reported_date <= as_of_date.

If you write code that uses fundamentals data indexed only by period_end without filtering by reported_date, you've created lookahead bias. This is a critical correctness bug in this domain.

Rule 2: Strategies Are Provider-Agnostic

Strategy code touches DataSource (the interface), never BloombergDataSource or YFinanceDataSource directly. The same strategy code must work with either implementation. If you find yourself importing a specific provider in strategy code, stop and refactor.

Rule 3: The Strategy Interface Is Stable

The Strategy ABC contract (rebalance_dates, target_weights, fit, params via Pydantic) is locked. Adding new strategies = implementing this interface. Modifying the interface itself requires explicit architectural change discussion, not a code change.

Rule 4: All Backtest Execution Goes Through the Job Queue

Even fast backtests. No sync HTTP endpoint that runs a backtest inline. Reason: uniform programming model, async-everywhere matches production patterns, prevents accidental long-blocking endpoints.

Rule 5: Reproducibility Fields Are Mandatory on Job Records

Every jobs row records: code_version (git SHA), dependencies_lock (pip-freeze), git_status_clean (bool), as_of_timestamp, explicit random seeds in params. Don't write code paths that create jobs without these fields populated.

Rule 6: Don't Add Containers Without Discussion

Container count is 7 in production. Adding an 8th container is an architectural decision, not an implementation choice. Tools that don't justify their own container should run as Python libraries inside an existing container.

Rule 7: Bloomberg Data Is Never in the Public Repo

Bloomberg-sourced data files, extracted Excel files, exported CSVs from BBG — none of these go in git. The .gitignore enforces this. If you're staging Bloomberg-derived data for commit, stop.

Rule 8: No User Accounts in v1 or v2

The project is open-access by design. Don't introduce auth tables, user models, JWT, OAuth, or session management. Admin endpoints get HTTP Basic Auth at Caddy, single shared credential. Rate-limiting is per-IP via slowapi.

File Layout (Authoritative)

azureus/                         # Python package root
  data/
    sources/                     # DataSource implementations
    schemas.py                   # Pydantic models for internal data
    db.py                        # SQLAlchemy session, query helpers
  features/
    base.py                      # Feature Protocol
    fundamentals/                # value, quality features
    market/                      # momentum, low_vol features
    macro/                       # (Phase 2)
    registry.py                  # central feature catalog
  models/
    base.py
    gbm.py                       # LightGBM wrapper
    cv/
      purged_kfold.py
      walk_forward.py
  strategies/
    base.py                      # Strategy ABC, StrategyParams, StrategyContext
    multi_factor_v1.py
    gbm_factors_v1.py
    benchmark.py
    registry.py
    docs/                        # methodology markdown per strategy
  backtesting/
    engine.py
    portfolio.py
    cost_model.py
    metrics.py
    results.py
  api/
    main.py                      # FastAPI app
    routes/
    schemas.py                   # Request/response Pydantic models
    rate_limit.py
  worker/
    tasks.py                     # RQ task definitions
  pipelines/                     # Prefect flows
    ingest_prices_free.py
    ingest_prices_bbg.py
    ...
  utils/
    dates.py                     # trading calendars, business days
    universe.py
    reproducibility.py           # git SHA capture, pip freeze

frontend/                        # Vite + React app
  src/
    components/
      ui/                        # shadcn primitives
      charts/                    # 8 Plotly chart components
      layout/
      strategy/
      backtest/
    pages/
    hooks/
    api/
    stores/                      # Zustand
    types/
    utils/

services/                        # per-service Dockerfiles + configs
  api/Dockerfile
  worker/Dockerfile
  base/Dockerfile                # shared Python base image
  caddy/Caddyfile
  postgres/                      # TimescaleDB image config
  prefect/
  mlflow/

alembic/                         # database migrations
  versions/
  env.py

docs/
  ARCHITECTURE.md                # the planning document
  RUNBOOK.md                     # operational procedures
  disaster-recovery.md

scripts/                         # one-off operational scripts
.github/workflows/               # CI/CD
docker-compose.yml
docker-compose.override.yml      # local-only overrides
docker-compose.prod.yml          # production overrides
pyproject.toml
.env.example
AGENTS.md                        # this file
README.md

When creating new files, place them in the right directory. When you can't tell where something belongs, ask.

Coding Standards

Python

• Type hints on every function signature. No exceptions for "trivial" functions.
• Pydantic for data shapes that cross module boundaries. Plain dataclasses or attrs are fine for internal-only.
• from __future__ import annotations at the top of every file. Forward references everywhere.
• Docstrings: Google style. Required on all public functions, classes, modules.
• No print() for logging. Use logging module with the project's logger config.
• No bare except:. Specify exception types.
• No # type: ignore without a comment explaining why.
• Async functions only where async actually matters (HTTP handlers, DB calls, Redis calls). Don't make compute-bound code async.

TypeScript

• Strict mode on. "strict": true in tsconfig.
• No any. If you genuinely need it, use unknown and narrow, or write a justifying comment.
• Component props typed. No implicit any on props.
• No barrel files (index.ts re-exports). Direct imports. Easier to grep, no circular import surprises.
• React.FC is not required. Just type props directly.

SQL

• No queries in route handlers. Route → service function → query function. Three layers minimum.
• Parameterized queries always. SQLAlchemy makes this automatic; if you write raw SQL, never string-format values in.
• EXPLAIN ANALYZE any new query that touches a hot path (backtest data loading, PIT fundamentals lookup).

Testing Standards

Required tests (don't merge without):

• PIT correctness test — every change to fundamentals query path must pass the AuditingDataSource lookahead regression test
• Migration tests — every Alembic migration must apply cleanly from empty schema and from previous version
• Cost model unit tests — any change to fees must include a test asserting expected bps on a known trade
• Strategy regression tests — Strategy 1 and Strategy 2 each have a "golden output" test that asserts a known backtest produces the same Sharpe / drawdown within tolerance

Skip these (intentionally low value):

• Trivial component render tests
• 100% coverage padding tests
• Tests that just assert constructors don't crash

Coverage targets:

• Core logic (backtester, features, cost model, CV): 90%
• API routes: path coverage on happy + error paths
• Frontend: ~50%, focused on logic-rich components

Common Mistakes to Avoid

These are things web tutorials get wrong for this domain. Don't propagate them.

1. Naive k-fold CV on time series. We use purged k-fold with embargo, expanding-window walk-forward. Not random shuffle.

2. Sharpe from monthly returns. We compute Sharpe from daily returns annualized. Monthly understates volatility.

3. Max drawdown as peak-to-trough. Wrong. We measure peak-to-recovery for duration.

4. Imputing missing fundamentals. v1 excludes tickers missing required features. No imputation. Documented as a methodological choice.

5. Using provider as a default parameter. Always required. Code that assumes a default provider is a bug waiting to happen.

6. Polling at sub-second intervals. 2 seconds. Faster doesn't help users and burns API resources.

7. Storing prices as FLOAT. NUMERIC(18, 6) always. Floats accumulate error.

8. Using datetime.now() in production code paths. Use injectable clock for testability. datetime.now() outside of logging is a code smell.

9. Catching exceptions and continuing silently in ingestion. Failures get logged to ingestion_runs with full context. No silent swallows.

10. Treating Bloomberg as a live data source. It's bulk extraction → Postgres. DataSource reads from our DB, never from Bloomberg APIs directly.

When You're About to Suggest Something New

Pause and check:

• Is it in docs/ARCHITECTURE.md? If no, ask before proceeding.
• Does it conflict with a "Hard Rule" above? If yes, stop.
• Would it add a container? If yes, ask.
• Would it require user accounts or auth beyond admin Basic Auth? If yes, stop.
• Would it require a new third-party service? If yes, ask.
• Is it on the "Don't suggest" column of the Stack table? If yes, stop.

Current Phase

Update this section as phases progress.

Current phase: Phase 0 public deployment live (monitoring polish still open) → Phase 1 complete (2026-05-23) → Phase 2 complete (2026-05-25) → Phase 3 complete (2026-05-25) → Phase 4 implementation in progress (2026-05-26).

Phase 4 implementation status (as of 2026-05-26): Branch work has started for the public ML path: public_free provider, Webull price ingestion, annual+quarterly yfinance fundamentals, gbm_factors_v1, purged k-fold, walk-forward LightGBM training, result diagnostics, read-only MLflow endpoints, generic frontend params, IC visualization, and MLflow Compose/Caddy wiring. Remaining production steps are adding real Webull secrets to local/prod env files, creating DNS A record mlflow -> 4.222.217.156, setting Caddy MLflow Basic Auth hash, upgrading the Azure VM to B2s, ingesting Webull coverage, and running the production Strategy 2 smoke.

Phase 3 status (as of 2026-05-25): Strategy 1 vertical slice is complete: feature registry, demo PIT fundamentals path, multi-factor strategy, FastAPI strategy/feature/backtest endpoints, RQ worker execution, persisted backtest_results, and Vite/TanStack Query frontend flow from strategy form to result dashboard. Browser-verified locally at http://localhost:5173 through FastAPI → RQ → Postgres. Production stack is live at https://azureus.tech; public smoke was verified through Caddy → FastAPI → RQ worker → Postgres, with frontend result rendering.

Phase 0 status (as of 2026-05-25): Local foundation done — public GitHub repo, two-job CI both green, local Docker Compose for Postgres+TimescaleDB and Redis (verified healthy), FastAPI /api/v1/health and /api/v1/version, Vite+React+TanStack Query frontend reading the health endpoint via Vite proxy. Production deployment is now live on Azure VM quantphemes-azure; azureus.tech DNS points to the VM and Caddy has issued valid TLS for azureus.tech and www.azureus.tech.

Azure VM deploy status (as of 2026-05-26): quantphemes-azure is reachable and runs the production compose stack at https://azureus.tech (Caddy → FastAPI → RQ worker → Postgres/TimescaleDB + Redis). Docker and 2 GiB swap are installed on the VM. Alembic migrations, ticker/universe seed data, ~900 days of yfinance prices, and demo yfinance fundamentals are loaded. DNS cutover and Caddy automatic TLS are complete. .github/workflows/deploy.yml deploys pushes to main after CI passes.

Phase 0 open items (must close before v0.1.0-foundation tag): optional shared services/base/Dockerfile, UptimeRobot monitor, final production screenshots.

Phase 0 success criterion (unchanged): A stranger visiting azureus.tech sees a working React page reading a live /api/v1/health. TLS valid. CI badge green. Push-to-main auto-deploys in <8 min.

Phase 1 entry condition: continuing into Phase 1 is an explicit deferral, not silent drift. The deployment items above reopen as soon as a host is selected; track them as carryover, do not let them quietly disappear.

Communication With the Developer

• The developer is doing this as both a portfolio piece and a learning exercise. Explain why, not just how, when relevant.
• The developer prefers direct technical feedback. If something is wrong, say so plainly.
• When the developer asks for help on a stuck problem, follow the 2-Day Rule from the architecture doc: ask whether the problem is a scope issue, an approach issue, or a knowledge gap. Different responses for each.
• Don't over-explain basics the developer already knows (advanced Python, React, SQL). Do explain things specific to this domain (PIT, purged CV, factor regression methodology).

End of AGENTS.md. Last updated: project planning lock (v1.0).