ILLATIO ECOSYSTEM // SPARSE MOE INFERENCE INFRASTRUCTURE GUILD CORE 140F6F7
Guild.
Menu
GitHub
Native inference / open source / v0.x

More model.
Your machine.

Your GPU is only part of the picture. Guild brings GPU memory, system RAM, and CPU compute together to run Mixture-of-Experts models locally.

Build your first run Explore the source ↗

C++ · CUDA · HIP / Windows & Linux setup paths

GUILD ENGINE // GPU VRAM + HOST RAM MOE PIPELINE ● VERIFIED SNAPSHOT
Architecture diagram showing token intake, GPU VRAM dense attention and Gated DeltaNet execution, Host RAM routed Mixture-of-Experts dispatch, and speculative verification. 01 // INTAKE & DRAFT Prompt / Drafter • Normal (T=1) • GPU MTP (k=1) • Lookup / Retr 02A // TIER 1 · GPU VRAM Dense & Attention State GQA / QSA + Gated DeltaNet Shared Expert + Top-K Router 02B // TIER 2 · SYSTEM RAM Sparse MoE Expert Pool 8/256 (Ornith) · 10/512 (Qwen) GPACK mmap + GEXEC AVX-512/AVX2 03 // VERIFY & COMMIT Window Verifier • Exact Target Dist • KV / GDN Commit • Cost min ms/tok
Ornith 35B W2 80.08–80.84 tok/s
Qwen W2 34.16–34.49 tok/s
Sampled VRAM 2.60–7.14 GiB (8G GPU)
Greedy Parity 100% Bitwise Exact
Heterogeneous MoE execution pipeline Conceptual diagram · measured metrics above
01 / Built for asymmetry

Not every weight.
Not all in VRAM.

In a sparse MoE model, each token routes through a subset of experts. Guild plans execution across the memory and compute available on your machine.

Conceptual routing / one layer

8 of 256 routed experts active per layer. Conceptual selection; not a live model trace.

  • GPU VRAM Dense path, attention state, cache, scratch
  • System RAM Routed expert complement & host KV
  • CPU compute Quantized expert execution
  • Mapped files Storage-backed expert tier

Designed for large-RAM servers, NUMA systems, and consumer GPUs paired with substantial host memory. The runtime finalizes its plan against live free VRAM.

Follow a token through the engine ↗
02 / Portable model. Local plan.

The model travels.
The tuning stays local.

GGUF Import weights GPACK Portable container GEXEC Machine-specific cache

GPACK

Model weights, architecture metadata, tokenizer assets, and optional expert data in a memory-mapped portable container.

Read the format ↗

GEXEC

Kernel choices, layouts, placement, and calibration bound to model and environment identity. A mismatch falls back to portable execution.

Read the contract ↗

Adaptive decoding

Choose among normal decoding, MTP, lookup, and session-local retrieval. Optimize time per committed token, with the target model verifying proposals.

Read the implementation report ↗

Core and these formats remain under development. Calibration combines measured microbenchmarks, estimates, and live session observations.

03 / Evidence over headlines

Performance is
a configuration.

Input-grounded speculative workloads can be much faster. Novel prose, short requests, and failed predictions can erase those gains. We publish the workload, hardware, strategy, and limitations alongside each selected result.

Explore measured results ↗
Measured server: 2× Xeon Gold 6130, 376 GiB system RAM, GTX 1070 8 GiB, Linux. RTX 5070 desktop results are pending.

Start close to the metal.

Read the guide. Check compatibility. Build for your hardware.

Find your model ↗