安装fairseq框架所需环境

Install Apex

git clone https://gitee.com/wingszb/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" ./

会报一些错误...解决过但是忘记了...

# 这个可以成功安装apex 但是要等待蛮久的  
conda install -c conda-forge nvidia-apex

Install Fairseq for Bart

pip uninstall -y enum34      # Prevent AttributeError: module 'enum' has no attribute 'IntFlag'
git clone --branch v0.9.0 https://gitee.com/salff/fairseq
mkdir ~/fairseq/models
cd ~/fairseq/models
wget 'https://dl.fbaipublicfiles.com/fairseq/models/bart.large.tar.gz'
tar -xzf bart.large.tar.gz
cd ~/fairseq
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json'
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe'
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt'

#Download encoder-updated.json file and put it under ~/fairseq
下载地址：https://github.com/amazon-research/BartGraphSumm/blob/main/data/encoder-updated.json

Install NLTK and Spacy:

pip install nltk spacy more_itertools
python -m spacy download en_core_web_sm
python -m nltk.downloader stopwords
python -m nltk.downloader punkt

Install ROUGE:(建议参考另一篇随笔）

sudo apt-get install -y cpanminus
cpanm —force XML::Parser

pip install -U git+https://gitee.com/sheldor369/pyrouge
git clone https://gitee.com/ginno/files2rouge
cd files2rouge
python setup_rouge.py
python setup.py install
pyrouge_set_rouge_path ~/.files2rouge

实在不行就clone整个环境吧

https://zhuanlan.zhihu.com/p/551614412

# 从一个服务器上发送压缩包到另一台服务器上
scp -P 222 ./fairseq_env.tar.gz mlgroup@IP_A:/home/mlgroup/miniconda3/envs/