深度学习优化器:《Lookahead Optimizer: k steps forward, 1 step back》
深度学习优化器:《Lookahead Optimizer: k steps forward, 1 step back》
项目地址:
https://github.com/michaelrzhang/lookahead
pytorch版本:
https://github.com/michaelrzhang/lookahead/blob/master/lookahead_pytorch.py
论文地址:
https://arxiv.org/abs/1907.08610
使用方法:(pytorch)
optimizer = # {any optimizer} e.g. torch.optim.Adam
if args.lookahead:
optimizer = Lookahead(optimizer, la_steps=args.la_steps, la_alpha=args.la_alpha)
We found that evaluation performance is typically better using the slow weights. This can be done in PyTorch with something like this in your eval loop:
if args.lookahead:
optimizer._backup_and_load_cache()
val_loss = eval_func(model)
optimizer._clear_and_load_backup()
@article{zhang2019lookahead,
title={Lookahead Optimizer: k steps forward, 1 step back},
author={Zhang, Michael R and Lucas, James and Hinton, Geoffrey and Ba, Jimmy},
journal={arXiv preprint arXiv:1907.08610},
year={2019}
}
本博客是博主个人学习时的一些记录,不保证是为原创,个别文章加入了转载的源地址,还有个别文章是汇总网上多份资料所成,在这之中也必有疏漏未加标注处,如有侵权请与博主联系。
如果未特殊标注则为原创,遵循 CC 4.0 BY-SA 版权协议。
posted on 2024-08-11 18:31 Angry_Panda 阅读(14) 评论(0) 编辑 收藏 举报