LLaMA 2

0 Introduction

What's new

  • Rotary Position Embedding (RoPE)
  • RMS Norm
  • Grouped Query Attention + KV Cache
  • SwiGLU

Diagram prospect

1 Model Architecture

1.1 Rotary Position Embedding

Paper: ROFORMER: ENHANCED TRANSFORMER WITH ROTARY POSITION EMBEDDING

f(q,m)f(k,n)=g(q,k,mn)
fq(q,m)fk(k,n)=[cosmθsinmθsinmθcosmθ]q[cosnθsinnθsinnθcosnθ]k

Euler's formula
eix=cosx+isinx
eimθ=cosmθ+isinmθ

QiR(iθ)=xiWQTR(iθ)=(ei+pi)WQTR(iθ)
KjR(jθ)=xjWKTR(jθ)=(ej+pj)WKTR(jθ)

fq(xi,i)fk(xj,j)=[QiR(iθ)][KjR(jθ)]T=[xiWQTR(iθ)][xjWKTR(jθ)]T=(ei+pi)WQTR(iθ)R(jθ)TWK(ej+pj)T=(ei+pi)WQTR(iθ)R(jθ)WK(ej+pj)T=(ei+pi)WQTR[(ij)θ]WK(ej+pj)T=g(xi,xj,ij)

1.2 RMS Norm

1.3 Grouped Query Attention + KV Cache

<1> Grouped Query Attention
GQA is the trade-off between Efficiency and Accuracy.

  • Efficiency: MHA < GQA < MQA
  • Accuracy: MHA > GQA > MQA

Figures from GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

<2> KV Cache

1.4 SwiGLU

SwiGLU means Swish(also refers to SiLU) and Gated Linear Unit, which is commonly used in the feed forward network of LLaMA 2, Mixtral 7B, Mixtral 8×7B.

import torch.nn as nn
import torch.nn.functional as F

class MLP(nn.Module):
    def __init__(self, config):
        self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size)
        self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size)
        self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size)
    def forward(self, x):
        hidden_states = self.down_proj(F.silu(self.gate_proj(x), dim = -1) * self.up_proj(x))
        return hidden_states

Reference

Video 1: Llama 2 模型结构解析 - CodeLearner | Bilibili
Blog 1: Llama 2详解 - CodeLearner | Zhihu
Blog 2: Understanding Llama2: KV Cache, Grouped Query Attention, Rotary Embedding and More
Video 2: Transformer的位置编码(Position Encoding)进展梳理
Blog 3: 二维旋转矩阵与向量旋转

posted @   ForHHeart  阅读(14)  评论(0编辑  收藏  举报
相关博文:
阅读排行:
· DeepSeek “源神”启动!「GitHub 热点速览」
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· 我与微信审核的“相爱相杀”看个人小程序副业
· C# 集成 DeepSeek 模型实现 AI 私有化(本地部署与 API 调用教程)
· spring官宣接入deepseek,真的太香了~
点击右上角即可分享
微信分享提示