gguf格式量化方法
gguf格式下，各种量化方法后的支持情况，及运行速度
| Library | CPU (AVX2) | CPU (ARM NEON) | Metal | cuBLAS | rocBLAS | SYCL | CLBlast | Vulkan | Kompute |
|---|---|---|---|---|---|---|---|---|---|
| K-quants | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ 慢 | ✅慢 | ✅ |
| I-quants | ✅慢 | ✅慢 | ✅慢 | ✅ | ✅ | Partial¹ | ✅ | ✅ | ✅ |
| Multi-GPU | N/A | N/A | N/A | ✅ | ❓ | ✅ | ❓ | ✅ | ❓ |
| K cache quants | ✅ | ❓ | ✅ | ✅ 慢 | Partial⁶慢 | ❓ | ✅ | ✅ | ✅ |
| MoE architecture | ✅ | ❓ | ✅ | ✅ | ✅ | ❓ | Partial² | ✅ | ✅ |
Note:
- ✅: Supported
- ❓: Not supported
- N/A: Not applicable
- Partial¹: Partially supported
- Partial²: Partially supported
- Partial⁶: Partially supported
- 🐢⁴: Limited support
- 🐢⁵: Limited support
- 🐢³: Limited support
