Navigation Page

Basic Notes for Coding

• ML / AI Algorithms from Scratch
• PyTorch Basic Notes
• DS / ML Basic Notes
• Python Basic Notes
• SQL Basic Notes
• Statistics Basic Notes
• LeetCode Notes
• LeetCode Hot 100

Web Development

• Build Lightweight AI SaaS: Next.js + Tailwind CSS
• Build Enterprise-Level AI SaaS: Vue + FastAPI & SQLAlchemy or Django
• Dev Env Config

Search & Recommendation & Advertisement

• Recommendation Systems Basic Notes

AI Agent

• Retrieval-Augmented Generation(RAG)

NLP & LLM

Text Input Preprocessing and Decode

• Tokenizer: BPE, WordPiece, and SentencePiece
• Text Representation: OneHot, BOW, N-grams, TF-IDF, Word2Vec, Glove, FastText, ELMO, BERT, SBERT
• Decode: Greedy Search, Beam Search, Top-K Sampling, Top-P Samping

Model Arcitecture

• RNNs & LSTMs & GRUs
• Transformer
• BERT
• LLAMA 2, Mixtral 8×7B(Mistral MoE)
• Multimodal Large Language Model(MLLM)
• Mamba

Pre-training

• Self-supervised Learning: AR, AE, and Contrastive Learning

Supervised Fine-tuning(SFT)

• Parameter-Efficient Fine-Tuning(PEFT): Prompt-Tuning, P-Tuning, Prefix-Tuning, LoRA, QLoRA, IA3
• Instruction Tuning

Model Preference Alignment

• Reinforcement Learning Basic Notes
• Reinforcement Learning from Human Feedback(RLHF): TRPO, PPO, DPO

Model Optimization

• Quantization & FLOPs: mixed-precision, fp16, bf16, int8, fp4, nf4
• Pruning
• Distill
• Distributed Training: DeepSpeed ZeRO 1/2/3 + Accelerate, Megatron-LM
• Inference Acceleration: ONNX Runtime, Xinference

Cloud Computing: Linux, Git, Docker and Deployment

• llama CPU version Inference
• LLM local deployment: Xinference, Ollama
• Git
• Docker

Summary

• Pre-trained Model Summary

Paper Reading Notes

• Momentum Contrast (MoCo) for Unsupervised Visual Representation Learning