Open Internet by MindsNet

lyogavin/airllm

Repository: lyogavin/airllm Stars: 29408 Forks: 3141 Primary language: Jupyter Notebook Discovery sources: trending:daily, trending:weekly Selection score: 78.0 Usefulness score: 8.5 Source confidence score: 5.0 Languages: Jupyter Notebook, Python, Shell Topics: chinese-llm, chinese-nlp, finetune, generative-ai, instruct-gpt, instruction-set, llama, llm, lora, open-models Summary: AirLLM 70B inference with single 4GB GPU README excerpt:  [**Quickstart**](#quickstart) | [**Configurations**](#configurations) | [**MacOS**](#macos) | [**Example notebooks**](#example-python-notebook) | [**FAQ**](#faq) **AirLLM** dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run **405B Llama 3.1** on **8GB**, **DeepSeek-V3 (671B)** on **~12GB**, and **Kimi K3 (2.8T)** — the largest open-source model released to date — on **under 4GB**, because sparse MoE models stream one expert at a time rather than a whole layer. <a href="https://github.com/lyogavin/airllm/stargazers"></a> [](https://pepy.tech/project/airllm) [](https://github.com/LianjiaTech/BELLE/blob/main/LICENSE) [](https://static.aicompose.cn/static/wecom_barcode.png?t=1671918938) [](https://discord.gg/2xffU5sn) [ ](https://pypi.org/project/airllm/) [![Website](https://img.shields.io/w

Computing & Technology, Computer Science, Artificial Intelligence