⚡ TPT Spark — Project Forge

A lean, native, cross-platform LLM runtime. No daemons. No HTTP overhead. No Python.

Load a .gguf model and start chatting — everything runs in a single binary, fully on-device.

Features

GPU inference via wgpu (Vulkan / Metal / DirectX 12) with custom WGSL compute shaders
CPU fallback via HuggingFace candle when no GPU adapter is found
Zero-copy mmap — model weights stream from disk directly into VRAM; no RAM copy
Real-time token streaming — tokens appear word-by-word as they are generated, with a multi-turn GPU KV cache that reuses the shared prefix between turns instead of recomputing it
Stop generation — cancel an in-flight inference at any time
Full sampling controls — temperature, top-p, top-k, min-p, seed, repeat/frequency/presence penalties, stop sequences
Conversation history — sessions persist to disk as JSON, can be resumed, and exported/imported
System prompt / persona — configurable per conversation
In-app model download — fetch GGUF files directly from any HTTPS URL
GPU adapter picker — enumerate and pin inference to a specific GPU on multi-GPU machines
Built-in benchmarking — warm-up + timed pass per model, tok/s and tok/s-per-GB, local history
Per-layer GPU profiling — optional detailed phase-by-phase timing of the wgpu inference pipeline
Headless mode — run without the GUI as a local JSON-RPC 2.0 server over a Unix socket / named pipe (see HEADLESS_API.md)
Fully local — no telemetry, no cloud, no network required after download
Single binary built with Rust + Tauri v2 — ~10 MB app overhead

Supported model architectures

Any GGUF model using standard blk.N.* tensor naming, quantized or full precision, including:

LLaMA / LLaMA 4 · Mistral / Mixtral · Qwen2 / 2.5 / 3 · Gemma / 2 / 3 / 4 · Phi-2 / 3 / 4 · GLM-4 · Command-R · StableLM · InternLM2 · DeepSeek V2/V3 · OLMo / OLMo2 · EXAONE · Granite · StarCoder2 · SOLAR · Baichuan / Baichuan2 · Grok · Falcon

See examples/README.md for known-good small models to start with, plus sample system prompts.

Quick start

1. Prerequisites

Requirement	Version
Rust	1.77+
Node.js	20+
Platform libs	Tauri prerequisites

2. Build

git clone https://github.com/tpt-solutions/tpt-spark
cd tpt-spark
npm install

# Dev mode — hot-reload frontend + Rust watch
npm run tauri dev

# Release binary — stub engine (UI smoke-test, no AI)
npm run tauri build

# Release binary — wgpu GPU engine (recommended for real inference)
npm run tauri build -- --features engine-wgpu

3. Add a model

Place a .gguf file and its matching tokenizer.json in the shared TPT models directory (used by every tool in the TPT suite, so a model downloaded here is immediately visible to tpt-crucible and tpt-gpu too):

~/.tpt/models/

(%USERPROFILE%\.tpt\models\ on Windows). Models from the old per-app directory are migrated automatically on first launch. Use the Browse button in the sidebar to point at a different directory instead.

Or use the Download Model panel in the app sidebar to fetch a GGUF directly by HTTPS URL.

Note: tokenizer.json must sit next to the .gguf file. Download it from the same HuggingFace model repository.

4. Chat

Open the app, select your model from the sidebar dropdown, click Load.
Weights upload to VRAM (GPU path) or RAM (CPU fallback) — a few seconds for a 4B model.
Type a message and press Enter to start chatting.
Click Stop to cancel generation at any time.

Engine feature flags

Cargo feature	Engine	Notes
`engine-stub` (default)	`StubEngine`	Echoes mock tokens — compiles everywhere, no native deps
`engine-candle`	`CandleEngine`	Real GGUF CPU inference via HuggingFace candle
`engine-wgpu`	`WgpuEngine`	GPU inference via wgpu + WGSL shaders; falls back to candle if no GPU

Architecture

Frontend (TypeScript + Vite)
    ↕  Tauri IPC (Channel — zero-copy token streaming)
Backend (Rust)
    ├── engine/mod.rs         LlmEngine trait + EngineHandle (Arc<Mutex<...>>)
    ├── engine/wgpu_engine    GPU path: mmap → VRAM → WGSL kernels
    ├── engine/wgpu_context   Adapter selection, device/queue, capability probing
    ├── engine/candle_engine  CPU path: HuggingFace candle GGUF
    ├── engine/cpu_fallback   Wraps candle_engine as the wgpu path's CPU fallback
    ├── engine/stub           Mock echo (default)
    ├── engine/shaders/       WGSL: gemm, attention, rope, rms_norm, silu, dequant
    ├── commands.rs           Tauri IPC commands
    ├── headless.rs           JSON-RPC 2.0 server (--headless / TPT_SPARK_HEADLESS=1)
    ├── conversation.rs       History persistence (JSON files)
    └── models/               GGUF directory scanner, shared ~/.tpt/models/ registry

Headless / scriptable mode

tpt-spark --headless

Starts the same engine behind a local JSON-RPC 2.0 socket instead of the GUI — useful for scripting or driving Spark from another TPT-suite tool. Full protocol and examples in HEADLESS_API.md.

Why not Ollama?

Ollama wraps llama.cpp in a Go HTTP daemon. TPT Spark replaces the HTTP layer with Tauri IPC and the Go daemon with a Rust process — cutting ~100 MB RAM overhead and the requirement for a running background service while keeping the same GGUF model support.

Contributing

Issues only — bug reports and feature requests are welcome via GitHub Issues; see CONTRIBUTING.md. Release binaries for Windows/macOS/Linux are built automatically by .github/workflows/release.yml.

License

Dual-licensed under either of

tpt-spark

Languages