conversation_qps

Imported from WuKongIM/WuKongIM (test/e2e/message/conversation_qps/AGENTS.md). Install upstream with npx skills add WuKongIM/WuKongIM --skill conversation_qps. Copyright stays with the author.

Conversation QPS release gate

This scenario owns fixed public HTTP load tests for /conversation/list and /conversation/sync on real single-node and three-node clusters with 256 hash slots.

Run: WK_E2E_CONVERSATION_QPS=1 WK_E2E_CONVERSATION_QPS_REPORT=/tmp/conversation-qps.json GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run TestConversationQPSReleaseGate -count=1 -timeout=18m -p=1 -v

Keep thresholds and dataset sizes in profile.json; do not add environment overrides. Prepare 600 generated groups through public APIs within five minutes. Use the existing authenticated /bench/v1/channel-runtime/evict endpoint to unload only those exact generated ranges on every node, respecting its busy-runtime guards. Preparation may wait at most one minute for busy work to settle; HTTP failures fail immediately. Do not restart processes or mix recovery traffic into the measurement. Require zero active runtimes across all roles before and after every phase and zero additional loads during warmup and measurement. No eviction runs during reads. Warm disk caches are intentional; this is not a physical cold-disk benchmark. Round-robin requests cover 24 users and all ingress nodes. Validate every response's exact persisted message identities. Use scheduled arrivals, bounded workers and a queue of max(workers, ceil(offered QPS * P99 budget in seconds)) entries. All scheduling/queue delay counts toward latency. Use no measured retries, and fail on drops, errors, incomplete pages, low QPS, high P99, runtime loads, membership writes, or missing metrics. Record CPU, heap and allocations; enforce the reviewed per-request allocation ceiling for each topology in addition to the QPS/P99 floor. Only non-Linux local diagnostics may record unavailable process CPU as null; Linux release runs require the CPU counter on every node.

The JSON report must include all 12 base cases, seven stress windows (eight endpoint results), and source/profile/binary identities. Also retain driver GOMAXPROCS, bounded host/driver counter snapshots around stress windows, per-scheduled-second timing/drop buckets and at most 16 slow samples per endpoint in the original release run. These observations do not change workload, workers, thresholds or enable profiling. Local dirty-tree reports are diagnostic only; publication requires clean exact-tag source evidence. Keep pure acceptance-policy tests in the default unit tier.

Capacity diagnosis is separate from publication: WK_E2E_CONVERSATION_CAPACITY=1 WK_E2E_CONVERSATION_CAPACITY_REPORT=/tmp/conversation-capacity/report.json GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSCapacity$' -count=1 -timeout=20m -p=1 -v. Reuse the exact persisted fixture. Diagnose the representative page-100 case for both endpoints and topologies; the release gate still covers all 12 page cases. Use 16 driver workers per node, ten-second bounded doubling probes (at most seven), one midpoint probe, and a fifteen-second confirmation. Report the confirmed lower bound and observed rejected rate, not an exact production capacity. Profile page-100 cases separately using the existing authenticated debug API, with bounded CPU/alloc profiles per node and a separate driver CPU profile. Profiled phases are excluded from capacity claims. Only queue drops, latency limits, HTTP 503 refusals and the exact observed legacy HTTP 400 head/recent-message backpressure and request-admission envelopes (pinned by unit tests) are expected overload, and each rejects that offered rate. Other HTTP/transport errors, malformed or incomplete successful responses and any runtime activation or membership mutation fail diagnosis.

For a local combined run, set WK_E2E_CONVERSATION_CAPACITY_WITH_GATE=1 alongside the release-gate environment and run TestConversationQPSReleaseGate with a 20-minute timeout. It runs the same six gate phases first on each topology, then two unprofiled capacity cases using the same prepared fixture. The report must contain 12 passing base phases, seven passing stress windows and four confirmed capacity cases. CI does not set this diagnostic flag and retains the fixed 18-minute gate deadline.

Throughput diagnosis is separate from the gate and capacity claims: WK_E2E_CONVERSATION_DIAGNOSIS=1 WK_E2E_CONVERSATION_DIAGNOSIS_REPORT=/tmp/conversation-diagnosis/report.json GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSDiagnosis$' -count=1 -timeout=35m -p=1 -v. It reuses the exact fixture and capacity staircase for page 100 on both topologies, then records three successive 60-second windows at each confirmed rate. Preserve refused windows as rejected evidence, never a stability pass; unexpected errors, mutations or runtime activation abort the diagnosis. Capture bounded existing per-node CPU/GC, hydration and RPC metrics around every window. CPU/allocation and five-second execution-trace captures run separately from steady load. Trace bodies are capped at 64 MiB per node and use only the existing authenticated debug API. Prefer Linux for CPU and wait attribution; do not compare absolute capacity across different operating systems as an optimization gain. The report records the actual binary, source, profile, driver and node GOMAXPROCS identities; complete diagnosis does not mean every load passed.

For an equal-load comparison to the 2026-09-12 Linux diagnosis, additionally set WK_E2E_CONVERSATION_DIAGNOSIS_FIXED_BASELINE=1. This skips capacity discovery and uses the frozen page-100 rates: single-node list/sync 800/240 QPS and three-node list/sync 1200/360 QPS. The receipt marks fixed_baseline_load=true; its rate fields describe offered comparison load, not newly confirmed capacity. The fixture, three-minute windows, metrics, profile bounds and failure semantics remain unchanged. This flag is diagnostic only and cannot alter the release gate.

WK_E2E_CONVERSATION_CAPACITY_LONG_CONFIRM=1 extends only the diagnostic capacity confirmation from 15 to 180 seconds, including each bounded halving retry. Each case records confirmation_seconds. Use a 40-minute timeout when combining it with the gate; the fixed publication phases and thresholds are unchanged. Preserve rejected attempts, even if a later lower rate passes.

WK_E2E_CONVERSATION_CAPACITY_FINE_SYNC=1 adds three ten-second page-100, three-node sync probes at 400, 420 and 440 QPS after the original capacity confirmation. Confirm passing candidates from highest to lowest for 180 seconds, stopping at the first pass. Store every probe and rejected confirmation under fine_attempts; only a passing long confirmation sets fine_confirmed_qps. The original attempts and confirmed baseline remain separate. Use a 45-minute timeout with the combined gate and long confirmation. This fixed diagnostic refinement never changes the release profile, admission limits or gate verdict.

Fixed backpressure attribution is opt-in: WK_E2E_CONVERSATION_BACKPRESSURE=1 WK_E2E_CONVERSATION_BACKPRESSURE_REPORT=/tmp/conversation-pressure/report.json WK_E2E_BINARY=/absolute/candidate GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSBackpressure$' -count=1 -timeout=18m -p=1 -v. Use Linux with process CPU metrics, three nodes, 48 driver workers, page 100 and the unchanged fixture. Fixed loads are list 1200 QPS, sync 400 QPS and sync 420 QPS, each in three successive 60-second windows, with no capacity search or retries. Run the exact same harness once per compared binary, and record any build overlays and hashes separately; source HEAD is only the checkout base, not proof of overlay contents. Record bounded per-node persisted-read histogram buckets and RPC/CPU metrics. Separate CPU/alloc/trace phases never count as capacity evidence. Driver scheduling/queue wait and HTTP-request percentiles are reported separately and cannot be added. Recognized refusals reject their window; all windows remain visible. Complete collection does not imply stable capacity, and it cannot change publication gates.

Peak confirmation uses the same attribution harness with two fixed unprofiled 180-second windows, list 1200 QPS and sync 420 QPS: WK_E2E_CONVERSATION_PEAK_CONFIRM=1 WK_E2E_CONVERSATION_BACKPRESSURE_REPORT=/tmp/conversation-peak/report.json WK_E2E_BINARY=/absolute/candidate GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSPeakConfirmation$' -count=1 -timeout=12m -p=1 -v. Run each exact variant three times with alternating/counterbalanced order, fresh clusters and the unchanged fixture, 48 driver workers and shared admission limit. Record binary hashes and any old-source overlay. Each endpoint window is uninterrupted and has no profiling, retries, rate halving or threshold overrides. The long_confirmation receipt distinguishes this from three one-minute windows. Recognized refusals remain rejected windows even when collection completes; unexpected failures, activation or writes abort. Run the unchanged release gate separately on the selected candidate. This is diagnostic, not publication evidence.

Sync-prefix comparisons may isolate the unchanged sync 420-QPS workload: WK_E2E_CONVERSATION_SYNC_PREFIX_CONFIRM=1 WK_E2E_CONVERSATION_BACKPRESSURE_REPORT=/tmp/conversation-prefix/report.json WK_E2E_BINARY=/absolute/candidate GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSSyncPrefixConfirmation$' -count=1 -timeout=9m -p=1 -v. This fixed preset sets sync_only=true and long_confirmation=true, retains the peak harness's fresh three-node fixture, 48 workers and uninterrupted 180-second window, and omits the unrelated list workload. Run three counterbalanced old/new pairs and the unchanged 12-case gate separately.

Mixed and hidden diagnosis is opt-in: WK_E2E_CONVERSATION_STRESS=1 WK_E2E_CONVERSATION_STRESS_REPORT=/tmp/conversation-stress/report.json WK_E2E_BINARY=/absolute/candidate GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSMixedAndHiddenDiagnosis$' -count=1 -timeout=25m -p=1 -v. The stress_diagnosis section in profile.json fixes a three-node, shared-start 600-second list-600/sync-200 window with 24 workers per endpoint (48 total). Endpoint latency/refusal results stay separate; CPU and allocations are recorded once for the shared window and must never be duplicated into endpoint costs. Then use public /conversations/delete calls to hide exact generated memberships, evict the fixture again, and measure first-page-100 and second-page-50 syncs at 20 QPS for 30 seconds each with 48 workers. Cohorts use alternating 50% hidden, 90% hidden, and exactly 99 visible then 100 hidden then one visible membership in activation/ID order. Verify exact ordered Channel IDs and every recent identity, including the legitimately empty second page of the 90%-hidden cohort. Setup mutations are outside measured windows. All reads must remain disk-only, with zero runtime loads/residency and zero membership writes. Recognized overload fails that window but remains in the diagnostic receipt; unexpected failures abort. Compare exact old/new binaries with the same harness. Thresholds for any release extension must be supported by completed diagnostic evidence.

WK_E2E_CONVERSATION_STRESS_STAGE=mixed or hidden isolates one fixed diagnostic stage on a fresh cluster; omission runs both. Preserve earlier rejected windows when adjusting error-envelope attribution; recognized refusals still fail. Publication always runs the stress_gate preset: mixed list 200/sync 60 QPS for 60 seconds, eight workers each; six hidden windows at 20 QPS for 15 seconds with eight workers. The v2 report binds this preset and requires all seven windows. Allocation ceilings apply to the whole shared window, divided by its total successful requests; never duplicate CPU or allocations into mixed endpoints.

Mixed release-load attribution is diagnostic only: WK_E2E_CONVERSATION_MIXED_ATTRIBUTION=1 WK_E2E_PRODUCT_SHA=<exact-product-commit> WK_E2E_BINARY=/absolute/product WK_E2E_CONVERSATION_MIXED_REPORT=/tmp/conversation-mixed/report.json GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestConversationQPSMixedAttribution$' -count=1 -timeout=12m -p=1 -v. Require four visible Linux AMD64 CPUs. Preserve three fixed 60-second windows using the unchanged release mixed preset and HTTP connection bounds. Record harness and product identities separately, retain per-scheduled-second timing/drop buckets and at most 16 slow arrival samples per endpoint, plus host/driver counters around each window, then capture bounded server allocations and simultaneous server and driver CPU profiles during a separate ten-second workload. Rejected windows remain evidence, unexpected failures or runtime/membership mutations abort, and collection completion never qualifies publication. The read-only conversation-qps-diagnose.yml verifies the optional exact prior binary digest.

Message-update same-host Linux diagnosis is opt-in: WK_E2E_MESSAGE_UPDATE_SOAK=1 WK_E2E_MESSAGE_UPDATE_REPORT=/tmp/message-update/report.json WK_E2E_BINARY=/absolute/product GOWORK=off go test -tags=e2e ./test/e2e/message/conversation_qps -run '^TestMessageUpdateThreeNodeSoak$' -count=1 -timeout=12m -p=1 -v. Use the unchanged 600-group fixture, three real processes, 256 Hash Slots, 12 physical Slots, three replicas, and node GOMAXPROCS=2. Preserve three 60-second mixed windows at the release mixed rates (list 200, sync 60 QPS), with the existing bounded driver and exact response checks. Compare the same compiled harness and resource limits against exact baseline/candidate binaries; record dirty source manifests and binary hashes. Set WK_E2E_MESSAGE_UPDATE_EDITED=1 only for the candidate: edit every tail once through public APIs before measurement, retaining 256-byte payloads and the original identities/order. Then measure changed-page and empty-page delta reads at 100 QPS for 60 seconds each, validating cursor coverage and edited contents. These fixed-cursor replays measure server read cost, not recommended SDK traffic. Collect bounded CPU and allocation profiles during a separate ten-second mixed load after the measured windows. Preserve rejected windows; completion is not acceptance. This diagnostic neither changes release thresholds nor establishes multi-host production capacity, online notification fanout, or sustained edit write capacity.

The delta diagnostic separately records selected-channel cold recovery: checking the retained original uses committed reads and may load that channel runtime. Warm exactly that channel before its steady windows, record the recovery time and load/residency changes, then require zero additional runtime loads and membership writes with stable residency during each delta window. Preserve the original zero-residency invariant for every conversation-only window. Evict fixture runtimes again only after delta measurement, before separate profiles.

For changed-delta CPU attribution, also set WK_E2E_MESSAGE_UPDATE_DELTA_PROFILE=1 with the edited diagnostic. Preserve the entire fixture, three mixed windows and both 60-second delta windows. Only the subsequent separate ten-second profiling workload changes to changed-delta 100 QPS with eight workers, retaining the already recovered channel. Mark this choice in the receipt; enforce exact edited content, zero errors/drops, no additional runtime loads or membership writes, and stable residency. Profiles remain bounded to eight CPU seconds and 32 MiB per body. Compare before/after binaries in counterbalanced order on fresh clusters, with at least three runs each and identical assigned resources; record all runs rather than selecting the fastest one. This is diagnostic evidence, not a release gate.