How long does the integration take?

Change one base URL in an OpenAI-compatible client, or install the Python or TypeScript SDK; the quickstart routes a first request in a few minutes.

Will this lock us into another vendor?

No. The router is Apache-2.0 open source, your provider keys stay yours, and the API is OpenAI-compatible, so leaving is a change of base URL.

What happens when a provider fails?

The router falls back to the next model your policy allows and records the fallback in the trace, so an outage costs a retry, not an incident.

tokie - Skill - OpenSmartRoute

Name: tokie
Author: feyninc

Imported from feyninc/tokie (AGENTS.md). Install upstream with npx skills add feyninc/tokie. Copyright stays with the author.

tokie

Rust tokenizer workspace for Chonkie. It loads Hugging Face tokenizer.json files and implements BPE, WordPiece, SentencePiece, and Unigram. The repo also contains Python bindings and a CLI.

License is MIT OR Apache-2.0.

Crates

Path	Package	Role
`crates/tokie`	`tokie` 0.1.4	Tokenizer library. Encode, decode, and load from Hugging Face.
`crates/pretokie`	`pretokie` 0.2.0	Pretokenizers used by tokie. Splits text into pieces before the model encodes them.
`crates/tokie-python`	`tokie-python`	PyO3 bindings. The Python module name is `tokie`.
`crates/tokie-cli`	`tokie-cli`	`tokie` binary for building, verifying, and uploading tokenizers.

The workspace members are crates/*. Project docs are README.md, docs/MIGRATION.md, and docs/TOKENIZERS.md.

Pretokie on this branch

This branch deleted the old crates/pretokie/src and examples/. Do not restore PretokConfig, Mask<C>, or Gpt2::new.

Each pretokenizer lives in its own file under src/impls/. Each pretokenizer's oracle tests live in their own file under crates/pretokie/tests/. Do not share a test file across pretokenizers. span.rs holds Span. src/walk/ is the shared walker: mod.rs is the kernel, and classify.rs emits seven masks for a 64-byte window (ASCII letter, digit, space, newline, other whitespace, apostrophe, and non-ASCII). The kernel inlines the classifier and the scheme's window. It does not know the split rules. Gpt2::count, Gpt2::count_upto, and Gpt2::walk are the calls. A pretokenizer is a zero-sized type. Callers pass text that is already normalized. Design notes, when present, are in notes/ and gitignored.

cargo check -p pretokie succeeds. tokie depends on pretokie 0.1.3 from crates.io until it is rewritten onto these cuts. Do not point that dependency at this crate.

Commands

cargo check -p pretokie
cargo test -p pretokie
cargo test -p tokie

Check or test one package. tokie builds against pretokie 0.1.3. This crate is 0.2.0.

Commit messages

Every commit subject uses one of these two prefixes:

feat(scope): short imperative summary
bug(scope): short imperative summary

feat is new behavior, a rewrite, docs, or a refactor. bug fixes incorrect behavior. Use no other prefix. That includes docs, chore, fix, and refactor.

scope is pretokie, tokie, python, cli, or docs. The summary is lowercase and imperative, with no trailing period. Add a body when the subject does not say why.

feat(pretokie): replace the config matrix with one file per formula

bug(tokie): keep a tab from prefixing the next gpt-2 piece

Commit only when the user asks.

Install tokie and it becomes one more candidate the router can pick - when it fits.

A listing is a routing target with a manifest: what it does, which domains it covers, what it costs and who publishes it. Once installed it sits beside your own models and tools, is scored like any other candidate for each request, and shows up in the trace when it wins. Ratings come from workspaces that installed it, one per account.

{ "ocm": "1", "id": "feyninc-tokie-tokie", "kind": "skill", "name": "tokie", "description": "Rust tokenizer workspace for [Chonkie](https://github.com/chonkie-inc/tokie). It loads Hugging Face `tokenizer.json` files and implements BPE, WordPiece, SentencePiece, and Unigram. The repo also contains Python bindings and a CLI.", "publisher": "feyninc", "version": "1.0.0", "capabilities": { "domains": [ "coding", "observability" ], "tags": [ "skill-md", "github-agents" ], "languages": [ "en" ] }, "quality_prior": 0.6, "examples": [ "Rust tokenizer workspace for [Chonkie](https://github.com/chonkie-inc/tokie). It loads Hugging Face `tokenizer.json` files and implements BPE, WordPiece, SentencePiece, and Unigram. The repo also contains Python bindings and a CLI." ], "primary": false, "metadata": { "source": { "provider": "github-agents", "repository": "https://github.com/feyninc/tokie", "path": "AGENTS.md", "ref": "36caddc410dfb68f50d613b36daa66417a2b969d", "url": "https://github.com/feyninc/tokie/blob/36caddc410dfb68f50d613b36daa66417a2b969d/AGENTS.md", "key": "feyninc/tokie/AGENTS.md" } }, "instructions": "# tokie\n\nRust tokenizer workspace for [Chonkie](https://github.com/chonkie-inc/tokie). It loads Hugging Face `tokenizer.json` files and implements BPE, WordPiece, SentencePiece, and Unigram. The repo also contains Python bindings and a CLI.\n\nLicense is MIT OR Apache-2.0.\n\n## Crates\n\n| Path | Package | Role |\n| --- | --- | --- |\n| `crates/tokie` | `tokie` 0.1.4 | Tokenizer library. Encode, decode, and load from Hugging Face. |\n| `crates/pretokie` | `pretokie` 0.2.0 | Pretokenizers used by tokie. Splits text into pieces before the model encodes them. |\n| `crates/tokie-python` | `tokie-python` | ", "cost": { "context_tokens": 690 } }

tokie

About

tokie

Crates

Pretokie on this branch

Commands

Commit messages

Install tokie and it becomes one more candidate the router can pick - when it fits.

What it costs to adopt - in engineering time.

One base URL

No vendor lock-in

Failover built in

Use it

Manifest

Reviews