The Lightweight OpenAI API Server

🔒 Local Inference Without Dependencies 🚀

Languages: 简体中文 · 繁體中文

Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.

🚀 If Shimmy helps you, consider sponsoring — 100% of support goes to keeping it free forever.

• $5/month: Coffee tier ☕ - Eternal gratitude + sponsor badge
• $25/month: Bug prioritizer 🐛 - Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 - Logo placement + monthly office hours
• $500/month: Infrastructure partner 🚀 - Direct support + roadmap input

🎯 Become a Sponsor | See our amazing sponsors 🙏

Shimmy is a single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

Under the hood it runs on Airframe, a pure-Rust WebGPU (WGSL) transformer engine built from scratch. No C++ toolchain, no backend flags, no compilation required. Version history lives in the CHANGELOG; see the Airframe CHANGELOG for engine release notes.

Why this matters:

• No C++ toolchain required — Rust only, top to bottom
• F32 precision throughout for deterministic, high-quality output
• WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated)
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX (see Extended Context)

11 model families · 25 certified model/quant combinations — every model below passes Shimmy's 5-gate GPU math verification pipeline (dequant, structural peel, numerical, decode≡prefill, logits) against the certification ledger. GGUF files load as-is; no recompilation, no hardcoded per-model constants.

• ⚡ TurboShimmy INT4 KV Cache — ~7× less KV VRAM with one flag (--kv-quant int4). Run Llama-3.2-3B on 4 GB GPUs.
• 🚀 OpenAI SDK Compatibility — drop-in replacement; VSCode Copilot, Cursor, Continue.dev, any OpenAI SDK.
• 🔧 Extended Context — YaRN RoPE scaling via SHIMMY_MAX_CTX.
• 📦 Migrating from v1.x — the llama.cpp backend was removed in v2.0; see the migration guide.
• 🧠 MOE support — Mixture-of-Experts CPU offloading is on the Airframe roadmap.
• 🏆 Certification — Every model passes a 5-gate mathematical verification pipeline. See docs/CERTIFICATION.md for how it works.

# 1) Download pre-built binary (Windows example)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe

# 2) Point it at a GGUF model
set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve &

# 3) See registered models
./shimmy list

# 4) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}' \
  | jq -r '.choices[0].message.content'

Full install, model acquisition, GPU, and VRAM sizing: docs/quickstart.md

Full documentation lives in docs/. Use this table to find what you need:

Shimmy maintains high code quality through comprehensive testing:

# Full test suite (default features = GPU engine)
cargo test --features airframe,huggingface

# Quick CPU-only tests (no GPU required)
cargo test --lib --no-default-features --features huggingface -- --test-threads=1

See docs/ppt-invariant-testing.md for technical details.

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 💝 Sponsorship: GitHub Sponsors

🌟 stars and climbing fast ⏱ <1s startup 🦀 100% Rust, no Python

🔥 Hacker News • Front Page Again • IPE Newsletter

Companies: Need invoicing? Email michaelallenkuykendall@gmail.com

MIT License - forever and always.

Philosophy: Infrastructure should be invisible. Shimmy is infrastructure.

Testing Philosophy: Reliability through comprehensive validation and property-based testing.

Forever maintainer: Michael A. Kuykendall Promise: This will never become a paid product Mission: Making local model inference simple and reliable