KTransformers
CPU-GPU hybrid inference and fine-tuning for large MoE language models
What it is
KTransformers runs and fine-tunes very large Mixture-of-Experts language models by splitting work between GPU and CPU, with AMX/AVX-optimized kernels, NUMA-aware memory handling, and INT4/INT8 or FP8 quantization. It lets models like DeepSeek-V3, Kimi-K2, and GLM run on a single consumer GPU paired with large system RAM, and integrates with SGLang for serving and LLaMA-Factory for LoRA, full-parameter, and DPO training. Researchers and hobbyists without datacenter GPU clusters use it to serve or adapt frontier-scale open models on their own hardware.