nanoGPT 逐行讲解——LayerNorm

LayerNorm第18-27行class LayerNorm(nn.Module):def __init__(self, ndim, bias):super().__init__()self.weight nn.Parameter(torch.ones(ndim))self.bias nn.Parameter(torch.zeros(ndim)) if bias else Nonedef forward(self, input):return F.layer_norm(input, self.weight.shape, self.weight, self.bias, 1e-5)作用层归一化让每层的输出分布更稳定关键点- weight 和 bias 是可学习参数- PyTorch 的 F.layer_norm 默认不支持 biasFalse这里做了封装- 1e-5 是 epsilon防止除零原文链接https://blog.csdn.net/lj1109053360/article/details/164012005