Pytorch在损失函数中为权重添加L1正则化

L1正则化可以使权重变稀疏，应用场景：对one-hot词袋模型中的词表进行裁剪时，根据权重weight筛选，此时需要权重越稀疏越好；

L1_Weight为超参数，可设定为1e-4

def train(model, iterator, optimizer, criteon):
    avg_acc, avg_loss = [], []
    model.train()     

    for batch in tqdm(iterator):
        text, label = batch[0].cuda(), batch[1].cuda()         

        pred = model(text)    
        l1_penalty = L1_Weight * sum([p.abs().sum() for p in model.fc.parameters()])
        loss = criteon(pred, label.long())        
        loss_with_penalty = loss + l1_penalty

        acc = utils.binary_acc(torch.argmax(pred.cpu(), dim=1), label.cpu().long())  
        avg_acc.append(acc)
        avg_loss.append(loss.item())

        optimizer.zero_grad()
        loss_with_penalty.backward()
        #loss.backward()
        optimizer.step()

    avg_acc = np.array(avg_acc).mean()
    avg_loss = np.array(avg_loss).mean()
    train_metrics = {'train_acc': avg_acc,
                     'train_loss': avg_loss
                     }
    logging.info(train_metrics)
    return avg_acc, avg_loss