1 bit AI 框架：Part 1.1，CPU 上的快速无损 BitNet b1.58 推理

(2024-10-23) 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs (1 bit AI 框架：Part 1.1，CPU 上的快速无损 BitNet b1.58 推理)

作者: Jinheng Wang; Hansong Zhou; Ting Song; Shaoguang Mao; Shuming Ma; Hongyu Wang; Yan Xia; Furu Wei;
期刊: （发表日期: 2024-10-23）
期刊分区:
本地链接: Wang 等 - 2024 - 1-bit AI Infra Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs.pdf
DOI: 10.48550/arXiv.2410.16144
摘要: Recent advances in 1-bit Large Language Models (LLMs), such as BitNet [WMD+23] and BitNet b1.58 [MWM+24], present a promising approach to enhancing the efficiency of LLMs in terms of speed and energy consumption. These developments also enable local LLM deployment across a broad range of devices. In this work, we introduce bitnet.cpp , a tailored software stack designed to unlock the full potential of 1-bit LLMs. Specifically, we develop a set of kernels to support fast and lossless inference of ternary BitNet b1.58 LLMs on CPUs. Extensive experiments demonstrate th