Imported from yamaru-eu/hardware-probe (
skills/expert-llmfit/SKILL.md). Install upstream withnpx skills add yamaru-eu/hardware-probe --skill expert-llmfit. Copyright stays with the author.
Expert LLMFit (BETA)
This skill provides the domain expertise required to maintain the LlmFitEngine inside remote-affiliate-api/src/core/llmfit.engine.ts and keep it synchronized with the upstream logic from https://github.com/AlexsJones/llmfit.
Core Concepts & Formulas
-
Inference is Memory-Bandwidth-Bound:
- Theoretical Max TPS =
Memory Bandwidth (GB/s) / Model Size (GB) - Real TPS Estimate =
Max TPS * Efficiency Factor (0.55)*RunMode Penalty. - The efficiency factor (0.55) accounts for kernel launch overhead and KV-cache reads.
- Theoretical Max TPS =
-
RunMode Penalties:
gpu: 1.0 (Full speed)moe_offload: 0.8 (Expert switching latency)cpu_gpu: 0.5 (Partial offload penalty)cpu_only: 0.3 (Fallback to slow RAM)
-
Memory Requirements (VRAM/RAM):
- Base Model Size =
Parameters * BytesPerParam- FP16/BF16: 2.0 bytes
- Q8: 1.0 byte
- Q4_K_M: 0.5 bytes (Standard for local inference recommendations)
- KV Cache Overhead: Context length requires additional VRAM.
- Rule of Thumb VRAM:
(Model Size in GB * 1.2)to account for context and OS overhead.
- Base Model Size =
-
Mixture of Experts (MoE):
- Fit/VRAM checks use Total Parameters.
- Speed/TPS estimates use Active Parameters (e.g., Mixtral 8x7B uses ~12.9B active params per token).
Verification & Adaptation Workflow
Whenever asked to "verify LLMFit logic" or "update the hardware recommendation engine", follow these steps:
Step 1: Upstream Sync
- Clone the upstream repository into a temporary directory:
mkdir -p /tmp/llmfit_external && git clone https://github.com/AlexsJones/llmfit /tmp/llmfit_external - Inspect key Rust files:
llmfit-core/src/fit.rs(for TPS and scoring formulas).llmfit-core/src/hardware.rs(for GPU bandwidth updates and compute capabilities).llmfit-core/src/models.rs(for Quantization multipliers and MoE active parameters).
Step 2: Codebase Audit (remote-affiliate-api)
- Compare the upstream formulas with our local TypeScript implementation in
src/core/llmfit.engine.ts. - Ensure the
estimateTokensPerSecondmethod correctly applies the0.55efficiency factor andRunModepenalties. - Verify
calculateRequiredMemoryproperly uses active vs total parameters.
Step 3: Database Metadata Alignment
- Check
prisma/schema.prismaandprisma/seed.ts. - Ensure
HardwareMetadatatable contains updatedbandwidthGBsfor new GPUs (e.g., RTX 50 series, M4 Apple Silicon). - Ensure
LlmModeltable correctly populatesactiveParametersBillionsfor MoE models during HuggingFace sync (src/services/huggingface.service.ts).
Step 4: Cleanup
- Remove the temporary upstream clone:
rm -rf /tmp/llmfit_external
Critical Rules
- DO NOT replace our TypeScript implementation with Rust code. We only port the mathematical logic and hardware constants.
- ALWAYS default to 4-bit quantization (
Q4_K_M) for consumer recommendation baselines. - NEVER recommend a model if its
minVramGbexceeds the user's availableHardwareMetadata.vramGb.
