Basic Notes for Coding
Interview Prep
- ML / AI Algorithms from Scratch
- LeetCode Notes
- LeetCode Hot 100
- Interview Experience for RecSys & Ads
Web Development
- Build Lightweight AI SaaS: Next.js + Tailwind CSS
- Build Enterprise-Level AI SaaS: Vue + FastAPI & SQLAlchemy or Django
- Dev Env Config
Search & RecSys & Ads
AI Agent
NLP & LLM
Text Input Preprocessing and Decode
- Tokenizer: BPE, WordPiece, and SentencePiece
- Text Representation: OneHot, BOW, N-grams, TF-IDF, Word2Vec, Glove, FastText, ELMO, BERT, SBERT
- Decode: Greedy Search, Beam Search, Top-K Sampling, Top-P Samping
Model Arcitecture
- RNNs & LSTMs & GRUs
- Transformer
- BERT
- LLAMA 2, Mixtral 8×7B(Mistral MoE)
- Multimodal Large Language Model(MLLM)
- Mamba
Pre-training
Supervised Fine-tuning(SFT)
- Parameter-Efficient Fine-Tuning(PEFT): Prompt-Tuning, P-Tuning, Prefix-Tuning, LoRA, QLoRA, IA3
- Instruction Tuning
Model Preference Alignment
Model Optimization
- Quantization & FLOPs: mixed-precision, fp16, bf16, int8, fp4, nf4
- Pruning
- Distill
- Distributed Training: DeepSpeed ZeRO 1/2/3 + Accelerate, Megatron-LM
- Inference Acceleration: ONNX Runtime, Xinference
Cloud Computing: Linux, Git, Docker and Deployment
Summary
Paper Reading Notes