LLM Inference

llama.cpp101.7k★

A server without a GPU. llama.cpp makes it possible. It runs quantized versions of open models (Llama, Mistral, Phi, Qwen, and dozens more) in pure C/C++ with optional GPU acceleration. No Python, no PyTorch, no CUDA dependency hell. Everything is free under MIT. No paid tier, no cloud, no account. Download a model file (GGUF format), point llama.cpp at it, and you're running inference. It includes a built-in HTTP server that exposes an OpenAI-compatible API, so your existing code that talks to GPT can talk to a local model with one URL change. The catch: you need hardware. A 7B parameter model needs ~4GB RAM (quantized). A 70B model needs ~40GB. Quality depends entirely on the model and quantization level; a heavily quantized model on a laptop won't match GPT-4. But for privacy-sensitive workloads, offline use, or just not wanting to pay per token, nothing else comes close.

vLLM75.3k★

VLLM is the fastest engine for serving them. It takes open-weight models and serves them over an OpenAI-compatible API, squeezing maximum throughput out of your GPUs. What's free: Everything. Apache 2.0 license. The entire inference engine, all optimizations (PagedAttention, continuous batching, tensor parallelism), the OpenAI-compatible API server. All free. vLLM's key innovation is PagedAttention, which manages GPU memory the way operating systems manage RAM, in pages instead of contiguous blocks. The result: 2-4x more throughput than naive inference. It's become the default serving engine for self-hosted LLMs. The catch: you need serious GPUs. Running a 70B parameter model requires 2-4 A100 GPUs ($1-2/hr on cloud, or $10K+ each to buy). Even a 7B model needs a decent GPU with 16GB+ VRAM. vLLM is free but the hardware is emphatically not. And it's optimized for NVIDIA GPUs. AMD ROCm support exists but is second-class.

text-generation-webui46.4k★

Text-generation-webui gives you a browser-based interface to do it. Load a model, chat with it, fine-tune it, generate images. It's the Swiss Army knife for local AI. The entire project is free under AGPL-3.0. Every feature (chat, notebook mode, model loading, LoRA training, multimodal/vision support, extensions) ships at $0. The developer sells some extension packs on Gumroad, but those are optional add-ons, not core features. Self-hosting is the only option, and the setup complexity depends on your GPU situation. If you have an NVIDIA card with 8GB+ VRAM, the one-click installers work well. AMD and Apple Silicon support exists but can be finicky. Expect 30-60 minutes for first-time setup including downloading a model. Solo developers: this is your playground. Run models locally, experiment with fine-tuning, keep your data private. Small teams: share a beefy GPU server running the API mode. Beyond that, look at dedicated inference servers like vLLM. The catch: GPU hardware requirements are real. You need a decent GPU to run anything useful. A 7B parameter model needs ~6GB VRAM. Anything bigger needs proportionally more. No GPU, no party.

LocalAI44.9k★

LocalAI runs your own AI models locally and exposes them through an OpenAI-compatible API. LLMs, image generation, speech-to-text: all from a single server. No cloud, no API keys, no data leaving your machine. MIT-licensed, free. Docker-based setup handles most of the complexity. A config file defines which models to load and which backends to use (llama.cpp, whisper, stable diffusion, and more). CPU inference is supported, which means any machine can run it. GPU acceleration is faster but not required. Models download at first startup. Developers who want to swap out OpenAI API calls with local models point their existing code at LocalAI's endpoint and change nothing else. Good for privacy-sensitive applications, air-gapped environments, and teams that want to control costs without changing application code. The catch: local inference is slower than cloud for most hardware setups. Model selection lags the frontier. You get privacy and cost control; you give up raw performance and convenience.

CLIProxyAPI23.4k★

CLIProxyAPI wraps existing AI coding CLIs, Gemini CLI, Claude Code, ChatGPT Codex, and others, and exposes them as OpenAI/Gemini/Claude-compatible API endpoints. The pitch is that you get access to models like Gemini 2.5 Pro and GPT-5 through their free CLI tiers, served as a standard API you can plug into any app. Let me be direct: this is a proxy that routes around pricing by using free CLI tools as backends, and exploding because free model access is irresistible. The homepage points to a subscription service at z.ai. The catch: this sits in a gray area. You're wrapping free CLI tools and serving them as APIs, which likely violates the terms of service for most of those CLIs. The sustainability of this approach depends entirely on providers not shutting it down. The MIT license covers the code, but the underlying model access is not yours to redistribute. Use at your own risk.

TensorRT-LLM13.3k★

TensorRT-LLM squeezes maximum inference performance out of NVIDIA GPUs for large language models. It handles quantization (FP8, FP4, INT4), custom attention kernels, paged KV caching, and multi-GPU deployment through a Python API. If you are serving LLMs at scale on NVIDIA hardware, this is the optimization layer that makes the economics work. Running it yourself means you need NVIDIA GPUs, full stop. No AMD, no Apple Silicon, no CPU fallback. You will also need CUDA installed and compatible driver versions. The setup is not trivial, but NVIDIA provides containers and Docker images that smooth out the worst of it. Once running, the performance gains over naive PyTorch inference are substantial, often 2-4x throughput improvements. For teams already committed to NVIDIA hardware, TensorRT-LLM is the right call over vLLM when you need every last token per second. vLLM is easier to set up and supports more hardware. llama.cpp is better for local, single-GPU experimentation. TensorRT-LLM is for production serving where GPU cost is a real line item. The catch: you are locked to NVIDIA forever. The library only works on their GPUs, and if your cloud costs push you toward AMD or custom silicon, you are rewriting your inference stack from scratch.

omlx8.6k★

Omlx puts an LLM inference server in your macOS menu bar. Click the icon, pick a model, and you have a local AI API running. It uses continuous batching (handles multiple requests efficiently) and SSD caching (models load faster after the first time) optimized specifically for Apple Silicon. This is the easiest way to run local LLMs on a Mac right now. No Docker, no Python environments, no config files. Menu bar app, one click, done. The API is OpenAI-compatible so any tool that talks to OpenAI can point at your local omlx instead. Apache 2.0 licensed, Python. The catch: Mac only. Apple Silicon specifically; Intel Macs are either unsupported or severely limited. The performance depends on your Mac's unified memory; 8GB will run small models, you need 32GB+ for anything serious. And 'menu bar simplicity' means less control over advanced settings like quantization, context length, and memory allocation.

flash-moe3.3k★

Flash-moe makes that possible. It uses a technique called Mixture of Experts (MoE) to run only the parts of the model that matter for each request, dramatically cutting the memory and compute needed. The pitch is simple: big model intelligence on small hardware. Models that normally need 32GB+ of VRAM can run on a laptop with 8-16GB of regular RAM. It's slower than running on a GPU, but it works. The catch: growing explosively but very early. The 'runs on a laptop' promise depends heavily on the model and your hardware. And MoE optimization is an active research area. Expect the approach to evolve fast.