嗯 vgpu 很重要
网速、硬盘杀手

配置

  • NVIDIA GeForce GTX 1060 5GB
  • WSL Debian Sid
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 531.41       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1060 5GB     On | 00000000:08:00.0  On |                  N/A |
|  0%   53C    P8               10W / 120W|    878MiB /  5120MiB |      6%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

MLC LLM

这个 他浏览器也能跑的 !!!

Phi-3.5-mini-instruct-q4f16_1-MLC

我这个显存试验了一圈 也就这个能跑起来 其余的 就 boom

安装

用的 cuad 12.1

这个需要的扩展包 nvidia-cuda-toolkit debian 在 sid 才提供了

先更改版本

1
2
rm /etc/apt/sources.list.d/debian.sources
echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ sid main contrib non-free non-free-firmware" > /etc/apt/sources.list

安装所需扩展

1
2
3
4
apt install nvidia-cuda-toolkit
apt install git git-lfs

git lfs install

再安装 conda

1
2
3
4
5
6
7
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" > /etc/apt/sources.list.d/conda.list

apt update
apt install conda

开跑

下载模型

1
2
3
4
5
6
cd /llm/mlc-llm

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mlc-ai/Phi-3.5-mini-instruct-q4f16_1-MLC

cd Phi-3.5-mini-instruct-q4f16_1-MLC
git lfs pull

配置环境

1
2
3
4
5
6
7
8
9
10
cd /llm/
conda create -n ai python=3.12
conda activate ai

# https://llm.mlc.ai/docs/install/mlc_llm.html#install-mlc-packages

python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu121 mlc-ai-nightly-cu121

# 测试
python -c "import mlc_llm; print(mlc_llm)"

开跑

可以体会到 什么叫做给出蹦字

1
2
3
4
cd /llm/
conda activate ai

mlc_llm chat Phi-3.5-mini-instruct-q4f16_1-MLC