2024 年 9月 7 日随笔档案 - ForHHeart

2024年9月7日

Distributed Training: DeepSpeed ZeRO 1/2/3 + Accelerate, Megatron-LM

摘要： 1 Introduction Github: https://github.com/microsoft/DeepSpeed ZeRO: Memory Optimizations Toward Training Trillion Parameter Models ZeRO-Offload: Democ 阅读全文

posted @ 2024-09-07 05:53 ForHHeart 阅读(1023) 评论(0) 推荐(0) 编辑

公告

昵称： ForHHeart
园龄： 2年9个月
粉丝： 0
关注： 0

+加关注

2025年3月

日

一

二

三

四

五

六

阅读排行榜

1. Distributed Training: DeepSpeed ZeRO 1/2/3 + Accelerate, Megatron-LM(1023)
2. Env Config(359)
3. Mamba(210)
4. Pre-trained Model Summary(184)
5. Quantization: fp16, bf16, int8, fp4, nf4(138)
6. Tokenizer: BPE, WordPiece, and SentencePiece(137)
7. Mixtral 8×7B (Mistral MoE)(89)
8. Navigation Page(53)
9. Python Basic Notes(52)
10. DS/MLE Key Competency and Occupational Classification Concluded from Job Descriptions(46)
11. transformers(45)
12. DS / ML Basic Notes(44)
13. ML / AI Algorithms from Scratch(36)
14. Build Lightweight AI SaaS: Next.js + Tailwind CSS(35)
15. Momentum Contrast (MoCo) for Unsupervised Visual Representation Learning(33)
16. Reinforcement Learning from Human Feedback(RLHF): TRPO, PPO, DPO(32)
17. LeetCode Notes in Python(32)
18. Transformer(27)
19. LeetCode Hot 100(22)
20. Multimodal Large Language Model(MLLM)(20)
21. Retrieval-Augmented Generation(RAG)(20)
22. Text Representation: OneHot, BOW, N-grams, TF-IDF, Word2Vec, Glove, FastText, ELMO, BERT, SBERT(18)
23. RecSys & Ads Basic Notes(18)
24. LLaMA 2(18)
25. SQL Basic Notes(18)
26. BERT(15)
27. Data Structures and Sort Algorithms Notes for Coding Interview(14)
28. PyTorch Basic Notes(13)
29. Self-supervised Learning(12)
30. LLM local deployment: Xinference, Ollama(12)
31. Python Web Dev(11)
32. RNNs & LSTMs & GRUs(11)
33. Loss Function(10)
34. Docker(10)
35. Interview Experience for RecSys & Ads(9)