Imported from waewoo/openwebui-kb-sync (AGENTS.md). Install upstream with npx skills add waewoo/openwebui-kb-sync. Copyright stays with the author.

AGENTS.md

This file provides guidance to Claude Code, Kilo Code, and other AI coding agents when working with code in this repository.

Project Overview

Docker Compose stack that automatically synchronizes GitLab repositories into the Knowledge Base (KB) of Open WebUI. Includes a provisioning system to configure LLM models, prompt templates, and Python tools via the Open WebUI API. Targeted at French-speaking teams (documentation and commit messages are in French).

Common Commands

All day-to-day operations go through make:

make setup            # First-time init: copy .env.example → .env, create dirs, chmod scripts
make install-deps     # Install Python dependency: pip install pyyaml (respects PIP_INDEX_URL)
make up               # Start the stack (removes orphaned containers)
make down             # Stop the stack
make restart          # Full stop + start
make sync-now         # Trigger sync immediately (no cron wait)
make logs-sync        # Tail sync script output (/var/log/sync-kb.log)
make logs-ofelia      # Tail Ofelia scheduler logs
make status           # Check container health
make provision        # Provision all: models + prompts + tools
make provision-models # Provision LLM models from config/models/*.yaml
make provision-prompts# Provision slash prompts from config/prompts/*.yaml
make provision-tools  # Provision Python tools from config/tools/*.yaml + *.py
make clean            # Remove containers and logs, keep volumes and repos/
make clean-all        # Full teardown including volumes (destructive)
make pull-images      # Pull Docker images (public or Artifactory registry)
make models-download  # Download HuggingFace models locally (requires internet, run once)
make models-check     # Verify local HF models are present

Architecture

Ofelia (cron: "0 23 * * *")
    └── executes scripts/sync-kb.sh inside openwebui container
            └── git clone / git pull --ff-only each repo
                    └── scripts/sync-kb-owui.py (incremental KB indexer)
                            └── SHA256 diff vs .kb-sync-state.json
                                    └── POST /api/v1/files/ + /api/v1/knowledge/{id}/file/add

config/ (git-versioned)
    └── kb-mapping.json, models/, prompts/, tools/
            └── make provision → scripts/provision-*.py → Open WebUI REST API

Services (docker-compose.yml):

• openwebui (image configurable via OPENWEBUI_IMAGE, port 3000→8080)
• ofelia (image configurable via OFELIA_IMAGE) — cron scheduler via Docker labels

sync-kb.sh — GitLab sync script (runs inside openwebui container):

• Array REPOS lists repo names to sync
• Each repo: git pull --ff-only if cloned, else git clone
• GitLab auth via token in URL: https://oauth2:${GITLAB_TOKEN}@${GITLAB_BASE_URL}/${REPO}.git
• After each git sync, calls sync-kb-owui.py --repo <name> for KB indexing
• Logs to /var/log/sync-kb.log; per-repo failures are non-fatal

sync-kb-owui.py — multi-mode KB indexer:

• --repo <name> : SHA256 differential sync of a cloned GitLab repo (called by sync-kb.sh)
• --file <path> --kb <name> : upload a single file
• --dir <path> --kb <name> : upload a full directory (no differential)
• --url <url> --kb <name> : scrape a web page, upload as .txt
• --text <content> --kb <name> : upload inline text content
• KB auto-created if it doesn't exist (resolved via config/kb-mapping.json for --repo mode)
• Skip silently if OPENWEBUI_API_KEY not set
• Zero external dependencies (Python stdlib only: argparse, urllib, hashlib, json, os, uuid)

config/kb-mapping.json — maps GitLab repo names to KB names:

{
  "kafka-config": "KB-Kafka",
  "terraform-config": "KB-Terraform"
}

Absent keys fall back to repo name.

Provisioning scripts (Python, run on host):

• provision-models.py — upserts via /api/v1/models
• provision-prompts.py — upserts via /api/v1/prompts
• provision-tools.py — upserts via /api/v1/tools
• All scripts are idempotent

Configuration

Copy .env.example to .env. Key variables:

Enterprise / Artifactory variables (disabled by default)

RAG Configuration

The openwebui service is tuned for configuration files (YAML, infra configs):

• Hybrid search: 40% BM25 keyword + 60% semantic vector
• Embedding model: sentence-transformers/all-MiniLM-L6-v2 (or Ollama if RAG_EMBEDDING_ENGINE=ollama)
• Reranking: BAAI/bge-reranker-v2-m3, top-50 → top-10
• Chunking: 1500 tokens / 200 overlap, relevance threshold 0.3

Enterprise HuggingFace Setup

When HuggingFace is not accessible (enterprise network):

Option A — Local models (recommended):

make models-download   # downloads to ./models/ (run once on a machine with internet)
make models-check      # verify models are present

Set in .env: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_MODELS_LOCAL_PATH=./models

Option B — Artifactory HF mirror: Set HF_ENDPOINT=https://artifactory.example.com/artifactory/api/huggingface

Option C — Ollama embeddings (zero HF dependency): Set RAG_EMBEDDING_ENGINE=ollama, RAG_EMBEDDING_MODEL=nomic-embed-text

Key Constraints

• Windows / Git Bash: Use printf instead of echo for ANSI colors in Makefile/scripts
• Python dependency: Provisioning requires pyyaml (make install-deps). python3 must be on PATH
• No test suite: Test with make sync-now + make logs-sync. Test provisioning with make provision-*
• Adding a repo: Edit REPOS array in scripts/sync-kb.sh. KB auto-created on first sync
• Changing sync schedule: Edit Ofelia label ofelia.job-exec.sync-kb.schedule in docker-compose.yml
• KB sync state: /opt/repos/.kb-sync-state.json — delete to force full re-index
• HF offline: Mount HF_MODELS_LOCAL_PATH as read-only volume to /root/.cache/huggingface/hub
• Commit messages: In French, atomic per file
• No external Python deps in sync scripts: stdlib only (argparse, urllib, hashlib, json, os, uuid)