transformers tokenizers需要glibc2.29解决办法
用conda安装新版transformers后，啪，报了个错
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /data/jcdu/miniconda3/envs/main/lib/python3.7/site-packages/tokenizers/tokenizers.cpython-37m-x86_64-linux-gnu.so)
解决方案：
- 安装旧版tokenizers卸了
- 用pip安装transformers和tokenizers
- 升级glibc，但是不小心就会干掉系统，还是罢了
吐槽两句，Ubuntu 18.04 LTS附带的glibc是2.27，这个版本至少还是一个很常用的版本，conda的glibc是自己提供的，非得干新版本，不知道是想做什么。同样的版本，pip就使用了旧版的glibc库，pip yyds！
关于这个糟糕的体验，在huggingface github上有详细的讨论（吐槽）
安装旧版
conda install -c huggingface transformers tokenizers=0.10.1
tokenizers 0.10.2开始就需要glibc2.29
tokenizers 0.10.1对应的transformers版本是4.11.3
顺便记录下ubuntu 18.04 能正常用的环境版本，conda env export > py36.yaml导出，conda env create -f py36.yaml
name: main
channels:
- pytorch
- huggingface
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- _openmp_mutex=5.1=1_gnu
- abseil-cpp=20211102.0=hd4dd3e8_0
- aiohttp=3.8.1=py37h7f8727e_1
- aiosignal=1.2.0=pyhd3eb1b0_0
- arrow-cpp=8.0.0=py37h60b952e_0
- async-timeout=4.0.1=pyhd3eb1b0_0
- asynctest=0.13.0=py_0
- attrs=21.4.0=pyhd3eb1b0_0
- aws-c-common=0.4.57=he6710b0_1
- aws-c-event-stream=0.1.6=h2531618_5
- aws-checksums=0.1.9=he6710b0_0
- aws-sdk-cpp=1.8.185=hce553d0_0
- blas=1.0=mkl
- boost-cpp=1.73.0=h7f8727e_12
- bottleneck=1.3.5=py37h7deecbd_0
- brotli=1.0.9=he6710b0_2
- brotlipy=0.7.0=py37h27cfd23_1003
- bzip2=1.0.8=h7b6447c_0
- c-ares=1.18.1=h7f8727e_0
- ca-certificates=2022.07.19=h06a4308_0
- certifi=2022.6.15=py37h06a4308_0
- cffi=1.15.0=py37hd667e15_1
- charset-normalizer=2.0.4=pyhd3eb1b0_0
- click=8.0.4=py37h06a4308_0
- cryptography=37.0.1=py37h9ce1e76_0
- cudatoolkit=11.3.1=h2bc3f7f_2
- dataclasses=0.8=pyh6d0b6a4_7
- datasets=2.4.0=py_0
- dill=0.3.4=pyhd3eb1b0_0
- ffmpeg=4.2.2=h20bf706_0
- filelock=3.6.0=pyhd3eb1b0_0
- freetype=2.11.0=h70c0345_0
- frozenlist=1.2.0=py37h7f8727e_0
- fsspec=2022.3.0=py37h06a4308_0
- gflags=2.2.2=he6710b0_0
- giflib=5.2.1=h7b6447c_0
- glog=0.5.0=h2531618_0
- gmp=6.2.1=h295c915_3
- gnutls=3.6.15=he1e5248_0
- grpc-cpp=1.46.1=h33aed49_0
- huggingface_hub=0.2.1=pyhd3eb1b0_0
- icu=58.2=he6710b0_3
- idna=3.3=pyhd3eb1b0_0
- importlib-metadata=4.11.3=py37h06a4308_0
- importlib_metadata=4.11.3=hd3eb1b0_0
- intel-openmp=2021.4.0=h06a4308_3561
- joblib=1.1.0=pyhd3eb1b0_0
- jpeg=9e=h7f8727e_0
- krb5=1.19.2=hac12032_0
- lame=3.100=h7b6447c_0
- lcms2=2.12=h3be6417_0
- ld_impl_linux-64=2.38=h1181459_1
- libboost=1.73.0=h28710b8_12
- libcurl=7.82.0=h0b77cf5_0
- libedit=3.1.20210910=h7f8727e_0
- libev=4.33=h7f8727e_1
- libevent=2.1.12=h8f2d780_0
- libffi=3.3=he6710b0_2
- libgcc-ng=11.2.0=h1234567_1
- libgfortran-ng=7.5.0=ha8ba4b0_17
- libgfortran4=7.5.0=ha8ba4b0_17
- libgomp=11.2.0=h1234567_1
- libidn2=2.3.2=h7f8727e_0
- libnghttp2=1.46.0=hce63b2e_0
- libopus=1.3.1=h7b6447c_0
- libpng=1.6.37=hbc83047_0
- libprotobuf=3.20.1=h4ff587b_0
- libssh2=1.10.0=h8f2d780_0
- libstdcxx-ng=11.2.0=h1234567_1
- libtasn1=4.16.0=h27cfd23_0
- libthrift=0.15.0=hcc01f38_0
- libtiff=4.2.0=h2818925_1
- libunistring=0.9.10=h27cfd23_0
- libvpx=1.7.0=h439df22_0
- libwebp=1.2.2=h55f646e_0
- libwebp-base=1.2.2=h7f8727e_0
- lz4-c=1.9.3=h295c915_1
- mkl=2021.4.0=h06a4308_640
- mkl-service=2.4.0=py37h7f8727e_0
- mkl_fft=1.3.1=py37hd3c417c_0
- mkl_random=1.2.2=py37h51133e4_0
- multidict=5.2.0=py37h7f8727e_2
- multiprocess=0.70.12.2=py37h7f8727e_0
- ncurses=6.3=h5eee18b_3
- nettle=3.7.3=hbbd107a_1
- numexpr=2.8.3=py37h807cd23_0
- numpy=1.21.5=py37h6c91a56_3
- numpy-base=1.21.5=py37ha15fc14_3
- openh264=2.1.1=h4ff587b_0
- openssl=1.1.1q=h7f8727e_0
- orc=1.7.4=h07ed6aa_0
- packaging=21.3=pyhd3eb1b0_0
- pandas=1.3.5=py37h8c16a72_0
- pillow=9.2.0=py37hace64e9_1
- pip=22.1.2=py37h06a4308_0
- protobuf=3.20.1=py37h295c915_0
- pyarrow=8.0.0=py37h992f0b0_0
- pycparser=2.21=pyhd3eb1b0_0
- pyopenssl=22.0.0=pyhd3eb1b0_0
- pyparsing=3.0.4=pyhd3eb1b0_0
- pysocks=1.7.1=py37_1
- python=3.7.13=h12debd9_0
- python-dateutil=2.8.2=pyhd3eb1b0_0
- python-xxhash=2.0.2=py37h7f8727e_0
- python_abi=3.7=1_cp37m
- pytorch=1.12.0=py3.7_cuda11.3_cudnn8.3.2_0
- pytorch-mutex=1.0=cuda
- pytz=2022.1=py37h06a4308_0
- pyyaml=6.0=py37h7f8727e_1
- re2=2022.04.01=h295c915_0
- readline=8.1.2=h7f8727e_1
- regex=2022.7.9=py37h5eee18b_0
- requests=2.28.1=py37h06a4308_0
- sacremoses=master=py_0
- scikit-learn=1.0.2=py37h51133e4_1
- scipy=1.7.3=py37hc147768_0
- setuptools=61.2.0=py37h06a4308_0
- six=1.16.0=pyhd3eb1b0_1
- snappy=1.1.9=h295c915_0
- sqlite=3.38.5=hc218d9a_0
- threadpoolctl=2.2.0=pyh0d69192_0
- tk=8.6.12=h1ccaba5_0
- tokenizers=0.10.1=py37_0
- torchaudio=0.12.0=py37_cu113
- torchvision=0.13.0=py37_cu113
- tqdm=4.64.0=py37h06a4308_0
- transformers=4.11.3=py_0
- typing-extensions=4.1.1=hd3eb1b0_0
- typing_extensions=4.1.1=pyh06a4308_0
- urllib3=1.26.9=py37h06a4308_0
- utf8proc=2.6.1=h27cfd23_0
- wheel=0.37.1=pyhd3eb1b0_0
- x264=1!157.20191217=h7b6447c_0
- xxhash=0.8.0=h7f8727e_3
- xz=5.2.5=h7f8727e_1
- yaml=0.2.5=h7b6447c_0
- yarl=1.6.3=py37h27cfd23_0
- zipp=3.8.0=py37h06a4308_0
- zlib=1.2.12=h7f8727e_2
- zstd=1.5.2=ha4553b6_0
prefix: /data/jcdu/miniconda3/envs/main
