Skip to content
OpenSmartRoute
Skillv1.0.0

inference-optimizer

Pick attention implementation, KV cache strategy, quantization, and speculative decoding for a new inference deployment. Use when you need help with inference optimizer.

by anubhavg-icpl(0) 0 installs
Free
Sign in to install

Free account. Installing gives you the manifest plus copy-paste snippets.

See reviews

About

Imported from anubhavg-icpl/vibe (skills/inference-optimizer/SKILL.md). Install upstream with npx skills add anubhavg-icpl/vibe --skill inference-optimizer. Copyright stays with the author (CC-BY-NC-SA-4.0).

Given an inference deployment (model name + params, target hardware, concurrency, max context length, latency SLO, throughput target), output:

  1. Serving stack. vLLM (default production), SGLang (lowest latency per token), TensorRT-LLM (NVIDIA optimal), llama.cpp (edge/CPU), MLX (Apple silicon). One-sentence reason.
  2. Attention implementation. Flash Attention 2 (Ampere/Ada default), Flash Attention 3 (Hopper), Flash Attention 4 (Blackwell, forward-only). Specify fallback.
  3. KV cache. Dtype (fp16 default, fp8 if supported), paged vs contiguous, prefix caching on/off, shared KV for parallel sampling.
  4. Quantization. fp16 / bf16 (default), int8 (weight-only), AWQ / GPTQ / GGUF for weights. Activation quantization only if benchmarked.
  5. Extra speedups. Speculative decoding (EAGLE 2 / Medusa / draft model), continuous batching (always on), chunked prefill (long-prompt workloads), prefix caching if repeated prompts.

Refuse to deploy Flash Attention 4 for training — it is forward-only at launch. Refuse to recommend fp8 KV cache without benchmarking quality impact on the target task. Flag any 70B+ model without GQA as having unmanageable KV cache at 32K+ context. Require prefix caching to be on for any agent/tool-calling deployment with repeated system prompts.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml
targets:
  - https://api.opensmartroute.ai/api/v1/registry/anubhavg-icpl-vibe-inference-optimizer/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

anubhavg-icpl-vibe-inference-optimizer.ocm.jsonjson
{
  "ocm": "1",
  "id": "anubhavg-icpl-vibe-inference-optimizer",
  "kind": "skill",
  "name": "inference-optimizer",
  "description": "Pick attention implementation, KV cache strategy, quantization, and speculative decoding for a new inference deployment. Use when you need help with inference optimizer.",
  "publisher": "anubhavg-icpl",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "transformers",
      "inference",
      "flash-attention",
      "kv-cache",
      "github"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Pick attention implementation, KV cache strategy, quantization, and speculative decoding for a new inference deployment. Use when you need help with inference optimizer."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "github",
      "repository": "https://github.com/anubhavg-icpl/vibe",
      "path": "skills/inference-optimizer/SKILL.md",
      "ref": "d51e7ac9a34111cd2e63daed83c2198f91570050",
      "url": "https://github.com/anubhavg-icpl/vibe/blob/d51e7ac9a34111cd2e63daed83c2198f91570050/skills/inference-optimizer/SKILL.md",
      "key": "anubhavg-icpl/vibe/skills/inference-optimizer/SKILL.md"
    },
    "license": "CC-BY-NC-SA-4.0"
  },
  "instructions": "Given an inference deployment (model name + params, target hardware, concurrency, max context length, latency SLO, throughput target), output:\n\n1. Serving stack. vLLM (default production), SGLang (lowest latency per token), TensorRT-LLM (NVIDIA optimal), llama.cpp (edge/CPU), MLX (Apple silicon). One-sentence reason.\n2. Attention implementation. Flash Attention 2 (Ampere/Ada default), Flash Attention 3 (Hopper), Flash Attention 4 (Blackwell, forward-only). Specify fallback.\n3. KV cache. Dtype (fp16 default, fp8 if supported), paged vs contiguous, prefix caching on/off, shared KV for parallel s",
  "cost": {
    "context_tokens": 319
  }
}

Fetch it by URL: GET /api/v1/registry/anubhavg-icpl-vibe-inference-optimizer/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.