lucene-s3-plugin

Imported from wxk6b1203/lucene-s3-plugin (AGENTS.md). Install upstream with npx skills add wxk6b1203/lucene-s3-plugin. Copyright stays with the author.

AGENTS.md

Project Overview

Lucene S3 is an experimental distributed search service: Lucene handles local writes and query execution, S3 (or compatible object storage) is the durable source for committed Lucene files, and etcd backs cluster state, node heartbeats, master lease, and shard routing. There are no primary/replica shards — each shard has a single write owner. The HTTP API is Elasticsearch-flavored.

Project Structure & Module Organization

Gradle multi-module Java project (root settings.gradle includes core, server, utility):

• utility/: shared utilities — JSON/YAML (Jackson), thread pools.
• core/: Lucene storage, S3/cache directory, metadata providers, cluster state, routing, search and indexing services.
• server/: CLI entry point (picocli), Vert.x HTTP API, request routing, distributed query/write coordination, Prometheus metrics.
• server/src/main/proto/lucene_s3/http/v1/http_api.proto: protobuf definitions; generated sources land in server/build/generated (regenerated by the build — never hand-edit generated code).
• versions.toml: Gradle version catalog — bump dependency versions here, not in module build files.
• test/: Python HTTP stress script (http_stress.py, stdlib only) and its docs; config/: YAML config examples (local overrides are gitignored and may contain credentials — never commit or copy them).
• generator/ contains only stale build output (unused; not part of the build).
• Tests live under each module's src/test/java. Runtime data, WAL, cache, logs, and distributions are build/runtime artifacts — do not commit (data/, logs/, build/).

Build, Test, and Development Commands

Requires JDK 25 (Gradle toolchain). On Windows, set JAVA_HOME to your JDK 25 install and prepend its bin to Path before using gradlew.bat.

.\gradlew.bat test --stacktrace              # all tests
.\gradlew.bat :core:test --stacktrace        # core tests only
.\gradlew.bat :server:test --stacktrace      # server/API tests only
.\gradlew.bat :core:test --tests "com.github.wxk6b1203.store.S3CachingDirectoryTest"   # single test class
.\gradlew.bat :server:run --args="server --http-port 9200 --data-path data/node-1"     # local single node
.\gradlew.bat :server:run --args="server --conf config/config.local.yml"               # run with YAML config
.\gradlew.bat :server:distZip                # build server/build/distributions/server-1.0-SNAPSHOT.zip
.\gradlew.bat :server:installDist            # unpacked distribution for quick iteration

On Linux/macOS use ./gradlew with shell quoting. The start scripts and :server:run automatically add --add-modules jdk.incubator.vector (required by Lucene's vectorized acceleration) — keep it when launching the jar manually.

Stress test against a running node:

python3 test/http_stress.py --base-url http://127.0.0.1:9200 --duration 10 --warmup-docs 100 \
  --write-workers 1 --read-workers 2 --knn-workers 1 --observe-workers 1

For etcd integration tests, set ETCD_TEST_ENDPOINTS=http://127.0.0.1:2379; those tests are gated with @EnabledIfEnvironmentVariable and skip silently otherwise.

Coding Style & Naming Conventions

• Java 25, 4-space indentation, UTF-8. Package ownership under com.github.wxk6b1203.
• Lombok is an annotation processor (compileOnly/annotationProcessor) in core and server.
• Prefer existing patterns before adding abstractions; keep comments short and only where they clarify non-obvious behavior.
• Public read preferences are weak and strong; owner and remote are internal implementation values.

Testing Guidelines

Tests use JUnit Jupiter. Name tests after behavior, e.g. nonMasterWriteReroutesStaleShardOwnerThroughMaster. Add focused tests for routing, metadata transitions, Lucene directory behavior, and HTTP APIs when changing those areas. Multi-node etcd tests must be gated with @EnabledIfEnvironmentVariable(named = "ETCD_TEST_ENDPOINTS", matches = ".+").

Commit & Pull Request Guidelines

Commit messages follow <type>(<scope>): <subject> with types feat|fix|docs|style|refactor|perf|test|chore|revert (e.g. fix(http): reroute stale shard owner writes). Subjects under 200 characters.

PRs should describe the behavior change, list verification commands, note etcd/S3 assumptions, and call out any migration or configuration impact.

Architecture Notes

Boundary rule: do not reintroduce primary/replica semantics. S3 is the durable source for committed Lucene files; each shard has exactly one current write owner. Cluster state and manifest metadata are etcd-backed in multi-node mode. Dependency chain: server → core → utility.

• Cluster state: ClusterState is the central immutable record. Single-node mode uses in-memory repository + noop coordinator (node is always master); with --etcd-endpoints set, EtcdClusterStateRepository/EtcdClusterCoordinator handle heartbeats and lease-based master election (CAS on the master key). Rebalance ticks are deduplicated by fingerprint (state version + sorted data-node IDs) with a 2× lease TTL fallback.
• Write fence: every write carries ownerTerm + allocationEpoch validated by the shard owner; forwarded writes carry x-lucene-s3-owner-term / x-lucene-s3-allocation-epoch headers. Deposed owners are rejected.
• Storage pipeline (WAL → S3): S3CachingDirectory tiers: local WAL dir (writes) → shared cache dir (reads, per-file locks + CRC32) → remote S3. File lifecycle DIRTY → UPLOADING → CLEAN → PINNED; S3 keys embed {fileName}.{crc32hex}.{size} so identical uploads are idempotent. RemoteObjectStore abstracts S3 (S3RemoteObjectStore, AWS SDK v2) with a local-file fallback for dev.
• Manifest: ManifestManager handles file metadata + async uploads; when given an external executor it does not close the pool (the canonical pool is owned by LuceneLocalShardIndexService). ManifestMetadataManager impls: etcd (per-key CAS) or in-memory when etcd is absent.
• Commit/refresh: IndexWriteOptions — commitEveryRequest=true (default) commits per request; otherwise commitAfterDocs/commitInterval thresholds checked by the background WRITE_MAINTENANCE tick. Refresh immediate (default) or interval (background). finishWrite() honors forceCommit/forceRefresh (force-merge always both). uploadWaitStrategy=wait_for_upload blocks commit until a clean snapshot is published or uploadWaitTimeout (default 30s).
• Search: weak (default) reads the local SearcherManager reader, may be stale under interval refresh; strong pins one clean remote snapshot generation per shard at plan time. Internal shard APIs add owner and remote read paths.
• Maintenance: ClusterMaintenanceService runs WRITE_MAINTENANCE, UPLOAD_RETRY, SNAPSHOT_GC, LIFECYCLE, CACHE_CLEANUP sequentially per tick; shard-level tasks take tryAcquireShardScope(shardId) to avoid racing on the same IndexWriter. ILM has active warm (owner-local force-merge, deduped per ownership epoch) and delete (master-only, sets deletePending) phases; hot/cold/frozen are accepted but no-ops. PITs pin snapshot generations against GC.
• Server layout: HttpApiServer delegates cluster introspection to ClusterIntrospectionHandlers; writes/mutations are forwarded (master or shard owner) via java.net.http.HttpClient (--http-forward-timeout). Internal shard endpoints live under /_internal/.... Prometheus metrics export on a separate --metrics-port.