Titan.
Production-inspired LLM inference runtime built from scratch in C++20 — tensor engine, GGUF loader, BPE tokenizer, CPU-optimized transformer inference. Zero prior C++ experience to a SIMD-optimized engine.
Titan is a production-inspired LLM runtime built completely from scratch, with the explicit goal of understanding every major subsystem that powers modern LLM inference — not by using an API, but by implementing the tensor math, memory management, and (eventually) request scheduling that sit underneath systems like ChatGPT. It is not a chatbot, RAG app, or LangChain wrapper — it's systems engineering: a tensor library, a GGUF model loader, a from-scratch BPE tokenizer, hand-rolled transformer forward pass, and a CPU-optimized execution engine (threading, SIMD, profiling). The project is explicitly split into two phases to keep scope realistic. V1 (target: end of August 2026) covers a CPU tensor engine, model loading, tokenization, transformer inference, and CPU optimization (threading, SIMD, profiling) — enough to run TinyLlama end-to-end on CPU with measured before/after performance deltas. V2 is the full production-serving layer: KV cache, request scheduler, continuous batching, an OpenAI-compatible API, and a live observability dashboard — architecturally identical in shape to systems like vLLM. A defining constraint of the project: it began with zero prior C++ experience. Every Milestone 1 issue is explicitly sequenced to build C++ fluency (RAII, ownership, templates, move semantics, threading) before Milestone 2 introduces performance optimization work. The repository is public and issue-driven from day one — commit history, PR descriptions, and issue writeups (each with problem statement, acceptance criteria, papers referenced, and the specific C++ concept it teaches) are written as a legible public record of that ramp-up, not just internal task tracking.
Problem
Most engineers who work with LLMs either consume them through an API (application layer) or train them (ML research layer). Very few understand the systems layer in between — how a forward pass actually executes on hardware, how memory is laid out and managed, how a request scheduler batches concurrent inference calls.
Solution
Build every layer of that stack from scratch, in order of dependency — tensor engine first, then model loading and tokenization, then a working CPU transformer, then optimization through profiling and measurement, then (V2) the scheduling and serving layer. Every optimization is benchmarked before/after.
- 01Two-phase PRD (V1/V2) scoping a 6-8 week CPU inference core against a full production-serving vision, with explicit success criteria and non-goals for each phase
- 02CMake + GoogleTest build system with GitHub Actions CI wired in from the first commit — tests pass on every push before any real subsystem exists
- 03MIT-licensed, fully public repository structured like a real open-source infra project: docs/architecture, docs/learning, docs/benchmarks, docs/profiling, issue-driven development
- 04Milestone 1 (CPU Runtime) broken into 8 sequenced, dependency-ordered issues (tensor storage → matmul → broadcasting → reshape/transpose → softmax/LayerNorm/GELU → BPE tokenizer → GGUF loader → wired-up TinyLlama generation)
- 05Learning track explicitly sequences C++ language concepts (RAII, ownership, templates, move semantics, threading, SIMD) against project milestones
- 4Milestones
- 8M1 Issues
- Aug 2026V1 Target
backend
- C++20
deployment
- GitHub Actions
other
- CMake
- GoogleTest
- Python