书生开源大模型训练营-第4讲-作业
基础作业：
构建数据集，使用 XTuner 微调 InternLM-Chat-7B 模型, 让模型学习到它是你的智能小助手，效果如下图所示，本作业训练出来的模型的输出需要将不要葱姜蒜大佬替换成自己名字或昵称！
微调前（回答比较官方） 
微调后（对自己的身份有了清晰的认知） 
---------------------------------------------------作业过程如下------------------------------------------
1、准备工作：创建工作目录ex4，创建虚拟环境，安装软件和依赖。
创建ex4虚拟环境：
conda create --name ex4 --clone=/root/share/conda_envs/internlm-base
激活ex4虚拟环境
conda activate ex4
创建工作目录ex4：
cd ~ && mkdir ex4 && cd ex4
从源码安装xtuner 0.1.9
mkdir xtuner_0.1.9 && cd xtuner_0.1.9 git clone -b v0.1.9 https://gitee.com/Internlm/xtuner
cd xtuner && pip install -e '.[all]'
屏幕输出：
(ex4) root@intern-studio-069640:~/ex4/xtuner_0.1.9# cd xtuner && pip install -e '.[all]' Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Obtaining file:///root/ex4/xtuner_0.1.9/xtuner Preparing metadata (setup.py) ... done Collecting bitsandbytes>=0.40.0 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9b/63/489ef9cd7a33c1f08f1b2be51d1b511883c5e34591aaa9873b30021cd679/bitsandbytes-0.42.0-py3-none-any.whl (105.0 MB) Collecting datasets Using cached https://pypi.tuna.tsinghua.edu.cn/packages/74/4d/63b033169534f0742b7fe13957118cae08c83b04bfde46511f397872e2e7/datasets-2.17.0-py3-none-any.whl (536 kB) Collecting einops Using cached https://pypi.tuna.tsinghua.edu.cn/packages/29/0b/2d1c0ebfd092e25935b86509a9a817159212d82aa43d7fb07eca4eeff2c2/einops-0.7.0-py3-none-any.whl (44 kB) Collecting fsspec<=2023.6.0 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e3/bd/4c0a4619494188a9db5d77e2100ab7d544a42e76b2447869d8e124e981d8/fsspec-2023.6.0-py3-none-any.whl (163 kB) Collecting lagent>=0.1.2 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/54/51/0cd9df1ec309b9d73e2a009bf61a8d8c84c34b27480994fe83a7fa8f24d3/lagent-0.2.1-py3-none-any.whl (69 kB) Collecting mmengine>=0.9.1 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/92/f8/0ec23b2d7fd2d3aebe05a70b8b4ff314c0cb552a614b1656ca1cb2a11633/mmengine-0.10.3-py3-none-any.whl (451 kB) Collecting modelscope Using cached https://pypi.tuna.tsinghua.edu.cn/packages/32/7f/5e49028db40c58a0ecea4f5a6ead189294353b793bb403d233b00cb35ac7/modelscope-1.12.0-py3-none-any.whl (5.6 MB) Collecting peft>=0.4.0 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/07/63/168af5aa8dbda9c23ad774a4c1d311cfe220c634e0d05a3a82a7cae01bd8/peft-0.8.2-py3-none-any.whl (183 kB) Collecting scipy Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f5/aa/8e6071a5e4dca4ec68b5b22e4991ee74c59c5d372112b9c236ec1faff57d/scipy-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.4 MB) Collecting SentencePiece Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7f/e5/323dc813b3e1339305f888d035e2f3725084fc4dcf051995b366dd26cc90/sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) Collecting tiktoken Using cached https://pypi.tuna.tsinghua.edu.cn/packages/16/05/5efbd91252ffb1301ea393d88ef736b33d41e75d4bcf0bd31d660050e400/tiktoken-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB) Requirement already satisfied: torch in /root/.conda/envs/ex4/lib/python3.10/site-packages (2.0.1) Collecting transformers<=4.34.0,>=4.32.1 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1a/d1/3bba59606141ae808017f6fde91453882f931957f125009417b87a281067/transformers-4.34.0-py3-none-any.whl (7.7 MB) Collecting transformers_stream_generator Using cached transformers_stream_generator-0.0.4-py3-none-any.whl Collecting deepspeed>=0.12.3 Using cached deepspeed-0.13.2-py3-none-any.whl Collecting mpi4py-mpich Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1a/e3/942a8e3322e3f1a265409d4028843c2770864f9ee699ba692296aa743232/mpi4py_mpich-3.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.0 MB) Collecting hjson (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1f/7f/13cd798d180af4bf4c0ceddeefba2b864a63c71645abc0308b768d67bb81/hjson-3.1.0-py3-none-any.whl (54 kB) Collecting ninja (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6d/92/8d7aebd4430ab5ff65df2bfee6d5745f95c004284db2d8ca76dcbfd9de47/ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB) Requirement already satisfied: numpy in /root/.conda/envs/ex4/lib/python3.10/site-packages (from deepspeed>=0.12.3) (1.26.2) Collecting packaging>=20.0 (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl (53 kB) Collecting psutil (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) Collecting py-cpuinfo (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl (22 kB) Collecting pydantic (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/db/dc/afecbd9650f486889181c6d1a0d675b580c06253ea7e304588e4c7485bdb/pydantic-2.6.1-py3-none-any.whl (394 kB) Collecting pynvml (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/5b/9c/adb8070059caaa15d5a572b66bccd95900d8c1b9fa54d6ecea6ae97448d1/pynvml-11.5.0-py3-none-any.whl (53 kB) Collecting tqdm (from deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/2a/14/e75e52d521442e2fcc9f1df3c5e456aead034203d4797867980de558ab34/tqdm-4.66.2-py3-none-any.whl (78 kB) Collecting arxiv (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/99/16/532c2aa4bc83b2356820efd4d1f619e45178dc3a0dc0cde16fbccdc43fc1/arxiv-2.1.0-py3-none-any.whl (11 kB) Collecting distro (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl (20 kB) Collecting func-timeout (from lagent>=0.1.2) Using cached func_timeout-4.3.5-py3-none-any.whl Collecting google-search-results (from lagent>=0.1.2) Using cached google_search_results-2.4.2-py3-none-any.whl Collecting griffe (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/aa/4c/7268d218ee38cb0e07d63fc3fe60fe19dc353f757db3d365f0b5ffba85be/griffe-0.40.1-py3-none-any.whl (116 kB) Collecting json5 (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/70/ba/fa37123a86ae8287d6678535a944f9c3377d8165e536310ed6f6cb0f0c0e/json5-0.9.14-py2.py3-none-any.whl (19 kB) Collecting jsonschema (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/39/9d/b035d024c62c85f2e2d4806a59ca7b8520307f34e0932fbc8cc75fe7b2d9/jsonschema-4.21.1-py3-none-any.whl (85 kB) Collecting jupyter (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/83/df/0f5dd132200728a86190397e1ea87cd76244e42d39ec5e88efd25b2abd7e/jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB) Collecting jupyter-client (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/43/ae/5f4f72980765e2e5e02b260f9c53bcc706cefa7ac9c8d7240225c55788d4/jupyter_client-8.6.0-py3-none-any.whl (105 kB) Collecting phx-class-registry (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9b/46/02f4f5fb40f5ccbb3fc23a328fb3314843375d050a3b40ec21a8c18b5762/phx_class_registry-4.1.0-py3-none-any.whl (13 kB) Requirement already satisfied: pillow in /root/.conda/envs/ex4/lib/python3.10/site-packages (from lagent>=0.1.2) (10.0.1) Collecting python-pptx (from lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/72/49/6eee83072983473e9905ffddd5c2032b9a0ca4616425560d6d582287b467/python_pptx-0.6.23-py3-none-any.whl (471 kB) Requirement already satisfied: requests in /root/.conda/envs/ex4/lib/python3.10/site-packages (from lagent>=0.1.2) (2.31.0) Collecting timeout-decorator (from lagent>=0.1.2) Using cached timeout_decorator-0.5.0-py3-none-any.whl Requirement already satisfied: typing-extensions in /root/.conda/envs/ex4/lib/python3.10/site-packages (from lagent>=0.1.2) (4.7.1) Collecting addict (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6a/00/b08f23b7d7e1e14ce01419a467b583edbb93c6cdb8654e54a9cc579cd61f/addict-2.4.0-py3-none-any.whl (3.8 kB) Collecting matplotlib (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c1/f2/325897d6c498278b0f8b460d44b516f5db865ddb4ba9018e9fe58a3e4633/matplotlib-3.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB) Collecting pyyaml (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/29/61/bf33c6c85c55bc45a29eee3195848ff2d518d84735eb0e2d8cb42e0d285e/PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) Collecting rich (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/be/be/1520178fa01eabe014b16e72a952b9f900631142ccd03dc36cf93e30c1ce/rich-13.7.0-py3-none-any.whl (240 kB) Collecting termcolor (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d9/5f/8c716e47b3a50cbd7c146f45881e11d9414def768b7cd9c5e6650ec2a80a/termcolor-2.4.0-py3-none-any.whl (7.7 kB) Collecting yapf (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/66/c9/d4b03b2490107f13ebd68fe9496d41ae41a7de6275ead56d0d4621b11ffd/yapf-0.40.2-py3-none-any.whl (254 kB) Collecting opencv-python>=3 (from mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d9/64/7fdfb9386511cd6805451e012c537073a79a958a58795c4e602e538c388c/opencv_python-4.9.0.80-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (62.2 MB) Collecting accelerate>=0.21.0 (from peft>=0.4.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1b/da/24a54b9205fce3bdbaad521c35944d0b0a2d292ac5ae921e484b76312b43/accelerate-0.27.2-py3-none-any.whl (279 kB) Collecting safetensors (from peft>=0.4.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d0/ba/b2254fafc7f5fdc98a2fa4d5a5eeb029fbf9589ec87f2c230c3ac0a1dd53/safetensors-0.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) Collecting huggingface-hub>=0.17.0 (from peft>=0.4.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/28/03/7d3c7153113ec59cfb31e3b8ee773f5f420a0dd7d26d40442542b96675c3/huggingface_hub-0.20.3-py3-none-any.whl (330 kB) Requirement already satisfied: filelock in /root/.conda/envs/ex4/lib/python3.10/site-packages (from torch) (3.13.1) Requirement already satisfied: sympy in /root/.conda/envs/ex4/lib/python3.10/site-packages (from torch) (1.11.1) Requirement already satisfied: networkx in /root/.conda/envs/ex4/lib/python3.10/site-packages (from torch) (3.1) Requirement already satisfied: jinja2 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from torch) (3.1.2) Collecting regex!=2019.12.17 (from transformers<=4.34.0,>=4.32.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/81/8a/96a62ce98e8ff1b16db56fde3debc8a571f6b7ea42ee137eb0d995cdfa26/regex-2023.12.25-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (773 kB) Collecting tokenizers<0.15,>=0.14 (from transformers<=4.34.0,>=4.32.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a7/7b/c1f643eb086b6c5c33eef0c3752e37624bd23e4cbc9f1332748f1c6252d1/tokenizers-0.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB) Collecting pyarrow>=12.0.0 (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d4/ca/ef67abb77f9dd51a0d3ff7fcebff58296068a046d7da352b9548070005ed/pyarrow-15.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (38.3 MB) Collecting pyarrow-hotfix (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e4/f4/9ec2222f5f5f8ea04f66f184caafd991a39c8782e31f5b0266f101cb68ca/pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB) Collecting dill<0.3.9,>=0.3.0 (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c9/7a/cef76fd8438a42f96db64ddaa85280485a9c395e7df3db8158cfec1eee34/dill-0.3.8-py3-none-any.whl (116 kB) Collecting pandas (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b3/b3/3102c3a4abca1093e50cfec2213102a1c65c0b318a4431395d0121e6e690/pandas-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB) Collecting xxhash (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/80/8a/1dd41557883b6196f8f092011a5c1f72d4d44cf36d7b67d4a5efe3127949/xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) Collecting multiprocess (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/bc/f7/7ec7fddc92e50714ea3745631f79bd9c96424cb2702632521028e57d3a36/multiprocess-0.70.16-py310-none-any.whl (134 kB) Collecting aiohttp (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/93/40/d3decda219ebd5410eba627601d537ec3782efbcadba308e9ce381cc0b71/aiohttp-3.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) Collecting attrs (from modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl (60 kB) Collecting gast>=0.2.2 (from modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/fa/39/5aae571e5a5f4de9c3445dae08a530498e5c53b0e74410eeeb0991c79047/gast-0.5.4-py3-none-any.whl (19 kB) Collecting oss2 (from modelscope) Using cached oss2-2.18.4-py3-none-any.whl Collecting python-dateutil>=2.1 (from modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Requirement already satisfied: setuptools in /root/.conda/envs/ex4/lib/python3.10/site-packages (from modelscope) (68.0.0) Collecting simplejson>=3.3.0 (from modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/cb/b6/ed513a0adc3e2c9654864ffb68266dcab5720d5653428d690e7e4fb32a6c/simplejson-3.19.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (137 kB) Collecting sortedcontainers>=1.5.9 (from modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) Requirement already satisfied: urllib3>=1.26 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from modelscope) (1.26.18) Collecting aiosignal>=1.1.2 (from aiohttp->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Collecting frozenlist>=1.1.1 (from aiohttp->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ec/25/0c87df2e53c0c5d90f7517ca0ff7aca78d050a8ec4d32c4278e8c0e52e51/frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB) Collecting multidict<7.0,>=4.5 (from aiohttp->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/33/62/2c9085e571318d51212a6914566fe41dd0e33d7f268f7e2f23dcd3f06c56/multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB) Collecting yarl<2.0,>=1.0 (from aiohttp->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/0ade1409d184cbc9e85acd403a386a7c0563b92ff0f26d138ff9e86e48b4/yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB) Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB) Collecting six>=1.5 (from python-dateutil>=2.1->modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB) Requirement already satisfied: charset-normalizer<4,>=2 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests->lagent>=0.1.2) (2.0.4) Requirement already satisfied: idna<4,>=2.5 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests->lagent>=0.1.2) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests->lagent>=0.1.2) (2023.11.17) INFO: pip is looking at multiple versions of tokenizers to determine which version is compatible with other requirements. This could take a while. Collecting tokenizers<0.15,>=0.14 (from transformers<=4.34.0,>=4.32.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/57/bd/45b5ef6b088880779f70acf60027f7043ca5fa1b98f4a4345cf3aea09044/tokenizers-0.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB) Collecting accelerate>=0.21.0 (from peft>=0.4.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e0/e5/20373eaee15adeb12872bc03355636c283cf3092fd7eb290bb974174b14e/accelerate-0.27.1-py3-none-any.whl (279 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c8/14/73c3d62e709c2ace755c826997b12f883f3cb6b138dec63ac1e2a68cd910/accelerate-0.27.0-py3-none-any.whl (279 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a6/b9/44623bdb05595481107153182e7f4b9f2ef9d3b674938ad13842054dcbd8/accelerate-0.26.1-py3-none-any.whl (270 kB) INFO: pip is still looking at multiple versions of tokenizers to determine which version is compatible with other requirements. This could take a while. Using cached https://pypi.tuna.tsinghua.edu.cn/packages/63/9c/c10fc10df1d4968406b3f3cffe5a7d9988a8583e3423fc4156d6c91ab62d/accelerate-0.26.0-py3-none-any.whl (270 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f7/fc/c55e5a2da345c9a24aa2e1e0f60eb2ca290b6a41be82da03a6d4baec4f99/accelerate-0.25.0-py3-none-any.whl (265 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/13/9e/ee987874058f2d93006961f6ff49e0bcb60ab9c26709ebe06bfa8707a4d8/accelerate-0.24.1-py3-none-any.whl (261 kB) INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C. Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d0/cf/364d550af711b5abe5129ac676896b223ba5a082d97fe400527a59c0c1f8/accelerate-0.24.0-py3-none-any.whl (260 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d9/92/2d3aecf9f4a192968035880be3e2fc8b48d541c7128f7c936f430d6f96da/accelerate-0.23.0-py3-none-any.whl (258 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/4d/a7/05c67003d659a0035f2b3a8cf389c1d9645865aee84a73ce99ddab16682f/accelerate-0.22.0-py3-none-any.whl (251 kB) Collecting transformers_stream_generator Using cached https://pypi.tuna.tsinghua.edu.cn/packages/bf/e8/785ec1627a60ca0ae7934525d2a24f419f146ff98b719f30ac76ced4fed4/transformers-stream-generator-0.0.3.tar.gz (12 kB) Preparing metadata (setup.py) ... done Collecting modelscope Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1d/1c/b40d3558879309e5b080e3f2eaaac016385487671508c362245bfd5e4cdf/modelscope-1.11.1-py3-none-any.whl (5.5 MB) Collecting datasets Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ec/93/454ada0d1b289a0f4a86ac88dbdeab54921becabac45da3da787d136628f/datasets-2.16.1-py3-none-any.whl (507 kB) Collecting dill<0.3.8,>=0.3.0 (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f5/3a/74a29b11cf2cdfcd6ba89c0cecd70b37cd1ba7b77978ce611eb7a146a832/dill-0.3.7-py3-none-any.whl (115 kB) Collecting datasets Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a0/93/da8a22a292e51ab76f969eb87bda8fd70cc3963b4dd71f67bb92a70a7992/datasets-2.16.0-py3-none-any.whl (507 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e2/cf/db41e572d7ed958e8679018f8190438ef700aeb501b62da9e1eed9e4d69a/datasets-2.15.0-py3-none-any.whl (521 kB) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/00/23/80a2147a547cb2fd59eb92a13787c849b3efaefcea02a5c963dfc93f7c56/datasets-2.14.7-py3-none-any.whl (520 kB) Collecting huggingface-hub>=0.17.0 (from peft>=0.4.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/aa/f3/3fc97336a0e90516901befd4f500f08d691034d387406fdbde85bea827cc/huggingface_hub-0.17.3-py3-none-any.whl (295 kB) Collecting feedparser==6.0.10 (from arxiv->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/92/1e/741fd94cf2855d251712868f2183cb6485a28daaa3947e1a7046dc036aca/feedparser-6.0.10-py3-none-any.whl (81 kB) Collecting sgmllib3k (from feedparser==6.0.10->arxiv->lagent>=0.1.2) Using cached sgmllib3k-1.0.0-py3-none-any.whl Collecting colorama>=0.4 (from griffe->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl (25 kB) Requirement already satisfied: MarkupSafe>=2.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from jinja2->torch) (2.1.1) Collecting jsonschema-specifications>=2023.03.6 (from jsonschema->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ee/07/44bd408781594c4d0a027666ef27fab1e441b109dc3b76b4f836f8fd04fe/jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB) Collecting referencing>=0.28.4 (from jsonschema->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/90/10/1c92edb0a0a14b67ff825bc338e74bc49ab27d3f3bae3f9a02838cba546f/referencing-0.33.0-py3-none-any.whl (26 kB) Collecting rpds-py>=0.7.1 (from jsonschema->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/15/f5/769fc90b3af55e6288ce683539ffd68b93dbdf1a5d86050f063828e5911e/rpds_py-0.18.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB) Collecting notebook (from jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/5f/38/f5a11c1e68bf3dbd54c7c98f301bf9495e8735803b42ee2f740c5b7c1ca5/notebook-7.1.0-py3-none-any.whl (5.0 MB) Collecting qtconsole (from jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d5/21/0887c50fa5bca7bfde29f65999a6ac234617f2a007b6b387aa4dc0ca36a8/qtconsole-5.5.1-py3-none-any.whl (123 kB) Collecting jupyter-console (from jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ca/77/71d78d58f15c22db16328a476426f7ac4a60d3a5a7ba3b9627ee2f7903d4/jupyter_console-6.6.3-py3-none-any.whl (24 kB) Collecting nbconvert (from jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c9/ec/c120b21e7f884a701e12a241992754e719adaf430d0d6b30c6655776bc35/nbconvert-7.16.0-py3-none-any.whl (257 kB) Collecting ipykernel (from jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/16/9a/0c7b514c73b42cf4ce516ee26c8940a0b23a9754dafaa459a939220240fd/ipykernel-6.29.2-py3-none-any.whl (116 kB) Collecting ipywidgets (from jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/70/1a/7edeedb1c089d63ccd8bd5c0612334774e90cf9337de9fe6c82d90081791/ipywidgets-8.1.2-py3-none-any.whl (139 kB) Collecting jupyter-core!=5.0.*,>=4.12 (from jupyter-client->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/86/a1/354cade6907f2fbbd32d89872ec64b62406028e7645ac13acfdb5732829e/jupyter_core-5.7.1-py3-none-any.whl (28 kB) Collecting pyzmq>=23.0 (from jupyter-client->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/67/bf/6bc0977acd934b66eacab79cec303ecf08ae4a6150d57c628aa919615488/pyzmq-25.1.2-cp310-cp310-manylinux_2_28_x86_64.whl (1.1 MB) Collecting tornado>=6.2 (from jupyter-client->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9f/12/11d0a757bb67278d3380d41955ae98527d5ad18330b2edbdc8de222b569b/tornado-6.4-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (435 kB) Collecting traitlets>=5.3 (from jupyter-client->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/45/34/5dc77fdc7bb4bd198317eea5679edf9cc0a186438b5b19dbb9062fb0f4d5/traitlets-5.14.1-py3-none-any.whl (85 kB) Collecting contourpy>=1.0.1 (from matplotlib->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/58/56/e2c43dcfa1f9c7db4d5e3d6f5134b24ed953f4e2133a4b12f0062148db58/contourpy-1.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (310 kB) Collecting cycler>=0.10 (from matplotlib->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl (8.3 kB) Collecting fonttools>=4.22.0 (from matplotlib->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a6/ba/5eac3e9c9bbc2dea3606e46de08bcef0908d74e7ccf89a71701b95a16747/fonttools-4.49.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB) Collecting kiwisolver>=1.3.1 (from matplotlib->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6f/40/4ab1fdb57fced80ce5903f04ae1aed7c1d5939dda4fd0c0aa526c12fe28a/kiwisolver-1.4.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) Collecting pyparsing>=2.3.1 (from matplotlib->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/39/92/8486ede85fcc088f1b3dba4ce92dd29d126fd96b0008ea213167940a2475/pyparsing-3.1.1-py3-none-any.whl (103 kB) INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while. Collecting multiprocess (from datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/35/a8/36d8d7b3e46b377800d8dec47891cdf05842d1a2366909ae4a0c89fbc5e6/multiprocess-0.70.15-py310-none-any.whl (134 kB) Collecting crcmod>=1.7 (from oss2->modelscope) Using cached crcmod-1.7-cp310-cp310-linux_x86_64.whl Collecting pycryptodome>=3.4.7 (from oss2->modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/af/20/5f29ec45462360e7f61e8688af9fe4a0afae057edfabdada662e11bf97e7/pycryptodome-3.20.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB) Collecting aliyun-python-sdk-kms>=2.4.1 (from oss2->modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/3d/ea/d88e08bfc4a0aee0111f1f24c98b19107bc6783441e7e944907c77b2243d/aliyun_python_sdk_kms-2.16.2-py2.py3-none-any.whl (94 kB) Collecting aliyun-python-sdk-core>=2.13.12 (from oss2->modelscope) Using cached aliyun_python_sdk_core-2.14.0-py3-none-any.whl Collecting pytz>=2020.1 (from pandas->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9c/3d/a121f284241f08268b21359bd425f7d4825cffc5ac5cd0e1b3d82ffd2b10/pytz-2024.1-py2.py3-none-any.whl (505 kB) Collecting tzdata>=2022.7 (from pandas->datasets) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/65/58/f9c9e6be752e9fcb8b6a0ee9fb87e6e7a1f6bcab2cdc73f02bb7ba91ada0/tzdata-2024.1-py2.py3-none-any.whl (345 kB) Collecting annotated-types>=0.4.0 (from pydantic->deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl (12 kB) Collecting pydantic-core==2.16.2 (from pydantic->deepspeed>=0.12.3) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/50/5e/2978d9f0e8d0cfd78e22115c028a41e0599e3d684e5aef7ed9bd18fcbd0c/pydantic_core-2.16.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) Collecting lxml>=3.1.0 (from python-pptx->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/25/5c/979167df4ca5a1c308105bb1590412c54bd1b0baa1883212f39cb42d4fcd/lxml-5.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.0 MB) Collecting XlsxWriter>=0.5.7 (from python-pptx->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f7/3e/05ba2194cd5073602422859c949a4f21310a3c49bf8dccde9e03d4522b11/XlsxWriter-3.1.9-py3-none-any.whl (154 kB) Collecting markdown-it-py>=2.2.0 (from rich->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/42/d7/1ec15b46af6af88f19b8e5ffea08fa375d433c998b8a7639e76935c14f1f/markdown_it_py-3.0.0-py3-none-any.whl (87 kB) Collecting pygments<3.0.0,>=2.13.0 (from rich->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/97/9c/372fef8377a6e340b1704768d20daaded98bf13282b5327beb2e2fe2c7ef/pygments-2.17.2-py3-none-any.whl (1.2 MB) Requirement already satisfied: mpmath>=0.19 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from sympy->torch) (1.3.0) Collecting importlib-metadata>=6.6.0 (from yapf->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c0/8b/d8427f023c081a8303e6ac7209c16e6878f2765d5b59667f3903fbcfd365/importlib_metadata-7.0.1-py3-none-any.whl (23 kB) Collecting platformdirs>=3.5.1 (from yapf->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/55/72/4898c44ee9ea6f43396fbc23d9bfaf3d06e01b83698bdf2e4c919deceb7c/platformdirs-4.2.0-py3-none-any.whl (17 kB) Collecting tomli>=2.0.1 (from yapf->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl (12 kB) Collecting jmespath<1.0.0,>=0.9.3 (from aliyun-python-sdk-core>=2.13.12->oss2->modelscope) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl (24 kB) Requirement already satisfied: cryptography>=2.6.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from aliyun-python-sdk-core>=2.13.12->oss2->modelscope) (41.0.3) Collecting zipp>=0.5 (from importlib-metadata>=6.6.0->yapf->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d9/66/48866fc6b158c81cc2bfecc04c480f105c6040e8b077bc54c634b4a67926/zipp-3.17.0-py3-none-any.whl (7.4 kB) Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich->mmengine>=0.9.1) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl (10.0 kB) Collecting comm>=0.1.1 (from ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6e/c1/e7335bd49aa3fa3bd453e34a4580b0076804f219897ad76d4d5aa4d8f22f/comm-0.2.1-py3-none-any.whl (7.2 kB) Collecting debugpy>=1.6.5 (from ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7a/27/78d5cf9c7aba43f8341e78273ab776913d2d33beb581ec39b65e56a0db77/debugpy-1.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB) Collecting ipython>=7.23.1 (from ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/fb/e7/07dc8b6541affd4de15f0e8fc855f238cb93d04c4f8490757226d12cdb5a/ipython-8.21.0-py3-none-any.whl (810 kB) Collecting matplotlib-inline>=0.1 (from ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f2/51/c34d7a1d528efaae3d8ddb18ef45a41f284eacf9e514523b191b7d0872cc/matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB) Collecting nest-asyncio (from ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl (5.2 kB) Collecting widgetsnbextension~=4.0.10 (from ipywidgets->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/99/bc/82a8c3985209ca7c0a61b383c80e015fd92e74f8ba0ec1af98f9d6ca8dce/widgetsnbextension-4.0.10-py3-none-any.whl (2.3 MB) Collecting jupyterlab-widgets~=3.0.10 (from ipywidgets->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/24/da/db1cb0387a7e4086780aff137987ee924e953d7f91b2a870f994b9b1eeb8/jupyterlab_widgets-3.0.10-py3-none-any.whl (215 kB) Collecting prompt-toolkit>=3.0.30 (from jupyter-console->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ee/fd/ca7bf3869e7caa7a037e23078539467b433a4e01eebd93f77180ab927766/prompt_toolkit-3.0.43-py3-none-any.whl (386 kB) Collecting beautifulsoup4 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b1/fe/e8c672695b37eecc5cbf43e1d0638d88d66ba3a44c4d321c796f4e59167f/beautifulsoup4-4.12.3-py3-none-any.whl (147 kB) Collecting bleach!=5.0.0 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ea/63/da7237f805089ecc28a3f36bca6a21c31fcbc2eb380f3b8f1be3312abd14/bleach-6.1.0-py3-none-any.whl (162 kB) Collecting defusedxml (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl (25 kB) Collecting jupyterlab-pygments (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b1/dd/ead9d8ea85bf202d90cc513b533f9c363121c7792674f78e0d8a854b63b4/jupyterlab_pygments-0.3.0-py3-none-any.whl (15 kB) Collecting mistune<4,>=2.0.3 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f0/74/c95adcdf032956d9ef6c89a9b8a5152bf73915f8c633f3e3d88d06bd699c/mistune-3.0.2-py3-none-any.whl (47 kB) Collecting nbclient>=0.5.0 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6b/3a/607149974149f847125c38a62b9ea2b8267eb74823bbf8d8c54ae0212a00/nbclient-0.9.0-py3-none-any.whl (24 kB) Collecting nbformat>=5.7 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f4/e7/ef30a90b70eba39e675689b9eaaa92530a71d7435ab8f9cae520814e0caf/nbformat-5.9.2-py3-none-any.whl (77 kB) Collecting pandocfilters>=1.4.1 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ef/af/4fbc8cab944db5d21b7e2a5b8e9211a03a79852b1157e2c102fcc61ac440/pandocfilters-1.5.1-py2.py3-none-any.whl (8.7 kB) Collecting tinycss2 (from nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/da/99/fd23634d6962c2791fb8cb6ccae1f05dcbfc39bce36bba8b1c9a8d92eae8/tinycss2-1.2.1-py3-none-any.whl (21 kB) Collecting jupyter-server<3,>=2.4.0 (from notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/25/d6/6ee093c967d11144aeb1b0b4952d30e51da8eb2737837ab612084c783a58/jupyter_server-2.12.5-py3-none-any.whl (380 kB) Collecting jupyterlab-server<3,>=2.22.1 (from notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ab/ac/a19c579bb8ab2a2aefcf47cd3787683e6e136378d7ab2602be3b8e628030/jupyterlab_server-2.25.3-py3-none-any.whl (58 kB) Collecting jupyterlab<4.2,>=4.1.1 (from notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/61/9b/8b974903425893806b15413fc899fefa78b0ed53e1699bcb8838c01a0ab2/jupyterlab-4.1.1-py3-none-any.whl (11.4 MB) Collecting notebook-shim<0.3,>=0.2 (from notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f9/33/bd5b9137445ea4b680023eb0469b2bb969d61303dedb2aac6560ff3d14a1/notebook_shim-0.2.4-py3-none-any.whl (13 kB) Collecting qtpy>=2.4.0 (from qtconsole->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7e/a9/2146d5117ad8a81185331e0809a6b48933c10171f5bac253c6df9fce991c/QtPy-2.4.1-py3-none-any.whl (93 kB) Collecting webencodings (from bleach!=5.0.0->nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl (11 kB) Requirement already satisfied: cffi>=1.12 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from cryptography>=2.6.0->aliyun-python-sdk-core>=2.13.12->oss2->modelscope) (1.16.0) Collecting decorator (from ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl (9.1 kB) Collecting jedi>=0.16 (from ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/20/9f/bc63f0f0737ad7a60800bfd472a4836661adae21f9c2535f3957b1e54ceb/jedi-0.19.1-py2.py3-none-any.whl (1.6 MB) Collecting stack-data (from ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl (24 kB) Collecting exceptiongroup (from ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b8/9a/5028fd52db10e600f1c4674441b968cf2ea4959085bfb5b99fb1250e5f68/exceptiongroup-1.2.0-py3-none-any.whl (16 kB) Collecting pexpect>4.3 (from ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl (63 kB) Collecting anyio>=3.1.0 (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/bf/cd/d6d9bb1dadf73e7af02d18225cbd2c93f8552e13130484f1c8dcfece292b/anyio-4.2.0-py3-none-any.whl (85 kB) Collecting argon2-cffi (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a4/6a/e8a041599e78b6b3752da48000b14c8d1e8a04ded09c88c714ba047f34f5/argon2_cffi-23.1.0-py3-none-any.whl (15 kB) Collecting jupyter-events>=0.9.0 (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e3/55/0c1aa72f4317e826a471dc4adc3036acd11d496ded68c4bbac2a88551519/jupyter_events-0.9.0-py3-none-any.whl (18 kB) Collecting jupyter-server-terminals (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7c/ec/ebb52454525e1d346bfa2ea91b3dcda3b92687bb73b2c25a6d621d9eeaf1/jupyter_server_terminals-0.5.2-py3-none-any.whl (13 kB) Collecting overrides (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/2c/ab/fc8290c6a4c722e5514d80f62b2dc4c4df1a68a41d1364e625c35990fcf3/overrides-7.7.0-py3-none-any.whl (17 kB) Collecting prometheus-client (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c7/98/745b810d822103adca2df8decd4c0bbe839ba7ad3511af3f0d09692fc0f0/prometheus_client-0.20.0-py3-none-any.whl (54 kB) Collecting send2trash>=1.8.2 (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a9/78/e4df1e080ed790acf3a704edf521006dd96b9841bd2e2a462c0d255e0565/Send2Trash-1.8.2-py3-none-any.whl (18 kB) Collecting terminado>=0.8.3 (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/69/df/deebc9fb14a49062a3330f673e80b100e665b54d998163b3f62620b6240c/terminado-0.18.0-py3-none-any.whl (14 kB) Collecting websocket-client (from jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1e/70/1e88138a9afbed1d37093b85f0bebc3011623c4f47c166431599fe9d6c93/websocket_client-1.7.0-py3-none-any.whl (58 kB) Collecting async-lru>=1.0.0 (from jupyterlab<4.2,>=4.1.1->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/fa/9f/3c3503693386c4b0f245eaf5ca6198e3b28879ca0a40bde6b0e319793453/async_lru-2.0.4-py3-none-any.whl (6.1 kB) Collecting httpx>=0.25.0 (from jupyterlab<4.2,>=4.1.1->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/39/9b/4937d841aee9c2c8102d9a4eeb800c7dad25386caabb4a1bf5010df81a57/httpx-0.26.0-py3-none-any.whl (75 kB) Collecting jupyter-lsp>=2.0.0 (from jupyterlab<4.2,>=4.1.1->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d4/35/8332e7a07f872324e29ae4620a41a21372a8dc710b63b873d80cb2184241/jupyter_lsp-2.2.2-py3-none-any.whl (68 kB) Collecting babel>=2.10 (from jupyterlab-server<3,>=2.22.1->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/0d/35/4196b21041e29a42dc4f05866d0c94fa26c9da88ce12c38c2265e42c82fb/Babel-2.14.0-py3-none-any.whl (11.0 MB) Collecting fastjsonschema (from nbformat>=5.7->nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9c/b9/79691036d4a8f9857e74d1728b23f34f583b81350a27492edda58d5604e1/fastjsonschema-2.19.1-py3-none-any.whl (23 kB) Collecting wcwidth (from prompt-toolkit>=3.0.30->jupyter-console->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/fd/84/fd2ba7aafacbad3c4201d395674fc6348826569da3c0937e75505ead3528/wcwidth-0.2.13-py2.py3-none-any.whl (34 kB) Collecting soupsieve>1.2 (from beautifulsoup4->nbconvert->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/4c/f3/038b302fdfbe3be7da016777069f26ceefe11a681055ea1f7817546508e3/soupsieve-2.5-py3-none-any.whl (36 kB) Collecting sniffio>=1.1 (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/5dba8ed157b0136607c7f2151db695885606968d1fae123dc3391e0cfdbf/sniffio-1.3.0-py3-none-any.whl (10 kB) Requirement already satisfied: pycparser in /root/.conda/envs/ex4/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.6.0->aliyun-python-sdk-core>=2.13.12->oss2->modelscope) (2.21) Collecting httpcore==1.* (from httpx>=0.25.0->jupyterlab<4.2,>=4.1.1->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/11/a6/24139fa27831cf2127fcf578d6d0a852a611f10cefecd800b1c557333d7a/httpcore-1.0.3-py3-none-any.whl (77 kB) Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.25.0->jupyterlab<4.2,>=4.1.1->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/95/04/ff642e65ad6b90db43e668d70ffb6736436c7ce41fcc549f4e9472234127/h11-0.14.0-py3-none-any.whl (58 kB) Collecting parso<0.9.0,>=0.8.3 (from jedi>=0.16->ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/05/63/8011bd08a4111858f79d2b09aad86638490d62fbf881c44e434a6dfca87b/parso-0.8.3-py2.py3-none-any.whl (100 kB) Collecting python-json-logger>=2.0.4 (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/35/a6/145655273568ee78a581e734cf35beb9e33a370b29c5d3c8fee3744de29f/python_json_logger-2.0.7-py3-none-any.whl (8.1 kB) Collecting rfc3339-validator (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7b/44/4e421b96b67b2daff264473f7465db72fbdf36a07e05494f50300cc7b0c6/rfc3339_validator-0.1.4-py2.py3-none-any.whl (3.5 kB) Collecting rfc3986-validator>=0.1.1 (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9e/51/17023c0f8f1869d8806b979a2bffa3f861f26a3f1a66b094288323fba52f/rfc3986_validator-0.1.1-py2.py3-none-any.whl (4.2 kB) Collecting ptyprocess>=0.5 (from pexpect>4.3->ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Collecting argon2-cffi-bindings (from argon2-cffi->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ec/f7/378254e6dd7ae6f31fe40c8649eea7d4832a42243acaf0f1fff9083b2bed/argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (86 kB) Collecting executing>=1.2.0 (from stack-data->ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/80/03/6ea8b1b2a5ab40a7a60dc464d3daa7aa546e0a74d74a9f8ff551ea7905db/executing-2.0.1-py2.py3-none-any.whl (24 kB) Collecting asttokens>=2.1.0 (from stack-data->ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/45/86/4736ac618d82a20d87d2f92ae19441ebc7ac9e7a581d7e58bbe79233b24a/asttokens-2.4.1-py2.py3-none-any.whl (27 kB) Collecting pure-eval (from stack-data->ipython>=7.23.1->ipykernel->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/2b/27/77f9d5684e6bce929f5cfe18d6cfbe5133013c06cb2fbf5933670e60761d/pure_eval-0.2.2-py3-none-any.whl (11 kB) Collecting fqdn (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/cf/58/8acf1b3e91c58313ce5cb67df61001fc9dcd21be4fadb76c1a2d540e09ed/fqdn-1.5.1-py3-none-any.whl (9.1 kB) Collecting isoduration (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7b/55/e5326141505c5d5e34c5e0935d2908a74e4561eca44108fbfb9c13d2911a/isoduration-20.11.0-py3-none-any.whl (11 kB) Collecting jsonpointer>1.13 (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/12/f6/0232cc0c617e195f06f810534d00b74d2f348fe71b2118009ad8ad31f878/jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB) Collecting uri-template (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e7/00/3fca040d7cf8a32776d3d81a00c8ee7457e00f80c649f1e4a863c8321ae9/uri_template-1.3.0-py3-none-any.whl (11 kB) Collecting webcolors>=1.11 (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d5/e1/3e9013159b4cbb71df9bd7611cbf90dc2c621c8aeeb677fc41dad72f2261/webcolors-1.13-py3-none-any.whl (14 kB) Collecting arrow>=0.15.0 (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f8/ed/e97229a566617f2ae958a6b13e7cc0f585470eac730a73e9e82c32a3cdd2/arrow-1.3.0-py3-none-any.whl (66 kB) Collecting types-python-dateutil>=2.8.10 (from arrow>=0.15.0->isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->jupyter->lagent>=0.1.2) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/28/50/8ed67814241e2684369f4b8b881c7d31a0816e76c8690ea8518017a35b7e/types_python_dateutil-2.8.19.20240106-py3-none-any.whl (9.7 kB) Installing collected packages: webencodings, wcwidth, timeout-decorator, sortedcontainers, sgmllib3k, SentencePiece, pytz, py-cpuinfo, pure-eval, ptyprocess, ninja, json5, hjson, func-timeout, fastjsonschema, crcmod, addict, zipp, xxhash, XlsxWriter, widgetsnbextension, websocket-client, webcolors, uri-template, tzdata, types-python-dateutil, traitlets, tqdm, tornado, tomli, tinycss2, termcolor, soupsieve, sniffio, six, simplejson, send2trash, scipy, safetensors, rpds-py, rfc3986-validator, regex, pyzmq, pyyaml, python-json-logger, pyparsing, pynvml, pygments, pydantic-core, pycryptodome, pyarrow-hotfix, pyarrow, psutil, prompt-toolkit, prometheus-client, platformdirs, phx-class-registry, pexpect, parso, pandocfilters, packaging, overrides, opencv-python, nest-asyncio, multidict, mpi4py-mpich, mistune, mdurl, lxml, kiwisolver, jupyterlab-widgets, jupyterlab-pygments, jsonpointer, jmespath, h11, gast, fsspec, frozenlist, fqdn, fonttools, feedparser, executing, exceptiongroup, einops, distro, dill, defusedxml, decorator, debugpy, cycler, contourpy, colorama, babel, attrs, async-timeout, async-lru, annotated-types, yarl, tiktoken, terminado, rfc3339-validator, referencing, qtpy, python-pptx, python-dateutil, pydantic, multiprocess, matplotlib-inline, markdown-it-py, jupyter-core, jedi, importlib-metadata, huggingface-hub, httpcore, griffe, google-search-results, comm, bleach, bitsandbytes, beautifulsoup4, asttokens, arxiv, argon2-cffi-bindings, anyio, aiosignal, yapf, tokenizers, stack-data, rich, pandas, matplotlib, jupyter-server-terminals, jupyter-client, jsonschema-specifications, httpx, deepspeed, arrow, argon2-cffi, aliyun-python-sdk-core, aiohttp, accelerate, transformers, mmengine, jsonschema, isoduration, ipython, aliyun-python-sdk-kms, transformers_stream_generator, peft, oss2, nbformat, ipywidgets, ipykernel, datasets, qtconsole, nbclient, modelscope, jupyter-events, jupyter-console, nbconvert, jupyter-server, notebook-shim, jupyterlab-server, jupyter-lsp, jupyterlab, notebook, jupyter, lagent, xtuner Running setup.py develop for xtuner Successfully installed SentencePiece-0.1.99 XlsxWriter-3.1.9 accelerate-0.27.2 addict-2.4.0 aiohttp-3.9.3 aiosignal-1.3.1 aliyun-python-sdk-core-2.14.0 aliyun-python-sdk-kms-2.16.2 annotated-types-0.6.0 anyio-4.2.0 argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 arrow-1.3.0 arxiv-2.1.0 asttokens-2.4.1 async-lru-2.0.4 async-timeout-4.0.3 attrs-23.2.0 babel-2.14.0 beautifulsoup4-4.12.3 bitsandbytes-0.42.0 bleach-6.1.0 colorama-0.4.6 comm-0.2.1 contourpy-1.2.0 crcmod-1.7 cycler-0.12.1 datasets-2.14.7 debugpy-1.8.1 decorator-5.1.1 deepspeed-0.13.2 defusedxml-0.7.1 dill-0.3.7 distro-1.9.0 einops-0.7.0 exceptiongroup-1.2.0 executing-2.0.1 fastjsonschema-2.19.1 feedparser-6.0.10 fonttools-4.49.0 fqdn-1.5.1 frozenlist-1.4.1 fsspec-2023.6.0 func-timeout-4.3.5 gast-0.5.4 google-search-results-2.4.2 griffe-0.40.1 h11-0.14.0 hjson-3.1.0 httpcore-1.0.3 httpx-0.26.0 huggingface-hub-0.17.3 importlib-metadata-7.0.1 ipykernel-6.29.2 ipython-8.21.0 ipywidgets-8.1.2 isoduration-20.11.0 jedi-0.19.1 jmespath-0.10.0 json5-0.9.14 jsonpointer-2.4 jsonschema-4.21.1 jsonschema-specifications-2023.12.1 jupyter-1.0.0 jupyter-client-8.6.0 jupyter-console-6.6.3 jupyter-core-5.7.1 jupyter-events-0.9.0 jupyter-lsp-2.2.2 jupyter-server-2.12.5 jupyter-server-terminals-0.5.2 jupyterlab-4.1.1 jupyterlab-pygments-0.3.0 jupyterlab-server-2.25.3 jupyterlab-widgets-3.0.10 kiwisolver-1.4.5 lagent-0.2.1 lxml-5.1.0 markdown-it-py-3.0.0 matplotlib-3.8.3 matplotlib-inline-0.1.6 mdurl-0.1.2 mistune-3.0.2 mmengine-0.10.3 modelscope-1.12.0 mpi4py-mpich-3.1.5 multidict-6.0.5 multiprocess-0.70.15 nbclient-0.9.0 nbconvert-7.16.0 nbformat-5.9.2 nest-asyncio-1.6.0 ninja-1.11.1.1 notebook-7.1.0 notebook-shim-0.2.4 opencv-python-4.9.0.80 oss2-2.18.4 overrides-7.7.0 packaging-23.2 pandas-2.2.0 pandocfilters-1.5.1 parso-0.8.3 peft-0.8.2 pexpect-4.9.0 phx-class-registry-4.1.0 platformdirs-4.2.0 prometheus-client-0.20.0 prompt-toolkit-3.0.43 psutil-5.9.8 ptyprocess-0.7.0 pure-eval-0.2.2 py-cpuinfo-9.0.0 pyarrow-15.0.0 pyarrow-hotfix-0.6 pycryptodome-3.20.0 pydantic-2.6.1 pydantic-core-2.16.2 pygments-2.17.2 pynvml-11.5.0 pyparsing-3.1.1 python-dateutil-2.8.2 python-json-logger-2.0.7 python-pptx-0.6.23 pytz-2024.1 pyyaml-6.0.1 pyzmq-25.1.2 qtconsole-5.5.1 qtpy-2.4.1 referencing-0.33.0 regex-2023.12.25 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rich-13.7.0 rpds-py-0.18.0 safetensors-0.4.2 scipy-1.12.0 send2trash-1.8.2 sgmllib3k-1.0.0 simplejson-3.19.2 six-1.16.0 sniffio-1.3.0 sortedcontainers-2.4.0 soupsieve-2.5 stack-data-0.6.3 termcolor-2.4.0 terminado-0.18.0 tiktoken-0.6.0 timeout-decorator-0.5.0 tinycss2-1.2.1 tokenizers-0.14.1 tomli-2.0.1 tornado-6.4 tqdm-4.66.2 traitlets-5.14.1 transformers-4.34.0 transformers_stream_generator-0.0.4 types-python-dateutil-2.8.19.20240106 tzdata-2024.1 uri-template-1.3.0 wcwidth-0.2.13 webcolors-1.13 webencodings-0.5.1 websocket-client-1.7.0 widgetsnbextension-4.0.10 xtuner-0.1.9 xxhash-3.4.1 yapf-0.40.2 yarl-1.9.4 zipp-3.17.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv (ex4) root@intern-studio-069640:~/ex4/xtuner_0.1.9/xtuner#
2、准备训练数据
创建训练数据集目录，并建立一个json文件，内容为训练数据：
mkdir -p /root/ex4/data && cd /root/ex4/data
生成训练数据的脚本genTrainSet.py：
import json # 输入你的名字 name = 'LittleSuccess' # 重复次数 n = 10000 data = [ { "conversation": [ { "input": "请做一下自我介绍", "output": "我是{}的小助手，内在是上海AI实验室书生·浦语的7B大模型哦".format(name) } ] } ] for i in range(n): data.append(data[0]) with open('train_set.json', 'w', encoding='utf-8') as f: json.dump(data, f, ensure_ascii=False, indent=4)
在data目录下运行该脚本：
python gen_trainset.py
屏幕输出：
(ex4) root@intern-studio-069640:~/ex4# mkdir -p /root/ex4/data && cd /root/ex4/data (ex4) root@intern-studio-069640:~/ex4/data# vi gen_trainset.py (ex4) root@intern-studio-069640:~/ex4/data# ls gen_trainset.py (ex4) root@intern-studio-069640:~/ex4/data# python gen_trainset.py (ex4) root@intern-studio-069640:~/ex4/data# ls gen_trainset.py train_set.json (ex4) root@intern-studio-069640:~/ex4/data# head -100 train_set.json [ { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { "input": "请做一下自我介绍", "output": "我是LittleSuccess的小助手，内在是上海AI实验室书生·浦语的7B大模型哦" } ] }, { "conversation": [ { (ex4) root@intern-studio-069640:~/ex4/data#
至此训练数据已经准备好。
3、基于基座模型进行lora微调
取得基座模型internlm 7b
mkdir -p /root/ex4/model/Shanghai_AI_Laboratory
cp -r /root/share/temp/model_repos/internlm-chat-7b /root/ex4/model/Shanghai_AI_Laboratory
拷贝配置文件到工作目录/root/ex4下：
xtuner copy-cfg internlm_chat_7b_qlora_oasst1_e3 .
修改训练配置：基座模型的位置、微调数据集的位置、微调的epoch数量、评估问题、微调数据集信息
启动微调：
xtuner train /root/ex4/internlm_chat_7b_qlora_oasst1_e3_copy.py
View Code
将微调得到的pth文件转换成HuggingFace的格式：
mkdir /root/ex4/hf export MKL_SERVICE_FORCE_INTEL=1 export CONFIG_NAME_OR_PATH=/root/ex4/internlm_chat_7b_qlora_oasst1_e3_copy.py export PTH=/root/ex4/work_dirs/internlm_chat_7b_qlora_oasst1_e3_copy/epoch_3.pth # pth文件转换为Hugging Face格式后参数存放的位置 export SAVE_PATH=/root/ex4/hf # 执行参数转换 xtuner convert pth_to_hf $CONFIG_NAME_OR_PATH $PTH $SAVE_PATH
(ex4) root@intern-studio-069640:~/ex4# xtuner convert pth_to_hf $CONFIG_NAME_OR_PATH $PTH $SAVE_PATH [2024-02-18 13:05:52,768] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library. Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it. [2024-02-18 13:06:33,380] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) quantization_config convert to <class 'transformers.utils.quantization_config.BitsAndBytesConfig'> Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:25<00:00, 3.24s/it] 02/18 13:07:12 - mmengine - INFO - dispatch internlm attn forward 02/18 13:07:12 - mmengine - WARNING - Due to the implementation of the PyTorch version of flash attention, even when the `output_attentions` flag is set to True, it is not possible to return the `attn_weights`. Load PTH model from /root/ex4/work_dirs/internlm_chat_7b_qlora_oasst1_e3_copy/epoch_3.pth Convert weights to float16 Saving HuggingFace model to /root/ex4/hf /root/.conda/envs/ex4/lib/python3.10/site-packages/peft/utils/save_and_load.py:148: UserWarning: Could not find a config file in /root/ex4/model/Shanghai_AI_Laboratory/internlm-chat-7b - will assume that the vocabulary was not modified. warnings.warn( All done!
把lora文件和基座模型merge起来：
export MKL_SERVICE_FORCE_INTEL=1 export MKL_THREADING_LAYER='GNU' # 原始模型参数存放的位置 export NAME_OR_PATH_TO_LLM=/root/ex4/model/Shanghai_AI_Laboratory/internlm-chat-7b # Hugging Face格式参数存放的位置 export NAME_OR_PATH_TO_ADAPTER=/root/ex4/hf # 最终Merge后的参数存放的位置 mkdir /root/ex4/hf_merge export SAVE_PATH=/root/ex4/hf_merge # 执行参数Merge xtuner convert merge \ $NAME_OR_PATH_TO_LLM \ $NAME_OR_PATH_TO_ADAPTER \ $SAVE_PATH \ --max-shard-size 2GB
屏幕输出：
(ex4) root@intern-studio-069640:~/ex4# export MKL_THREADING_LAYER='GNU' (ex4) root@intern-studio-069640:~/ex4# (ex4) root@intern-studio-069640:~/ex4# # 原始模型参数存放的位置 (ex4) root@intern-studio-069640:~/ex4# export NAME_OR_PATH_TO_LLM=/root/ex4/model/Shanghai_AI_Laboratory/internlm-chat-7b (ex4) root@intern-studio-069640:~/ex4# (ex4) root@intern-studio-069640:~/ex4# # Hugging Face格式参数存放的位置 (ex4) root@intern-studio-069640:~/ex4# export NAME_OR_PATH_TO_ADAPTER=/root/ex4/hf (ex4) root@intern-studio-069640:~/ex4# (ex4) root@intern-studio-069640:~/ex4# # 最终Merge后的参数存放的位置 (ex4) root@intern-studio-069640:~/ex4# mkdir /root/ex4/hf_merge (ex4) root@intern-studio-069640:~/ex4# export SAVE_PATH=/root/ex4/hf_merge (ex4) root@intern-studio-069640:~/ex4# (ex4) root@intern-studio-069640:~/ex4# # 执行参数Merge (ex4) root@intern-studio-069640:~/ex4# xtuner convert merge \ > $NAME_OR_PATH_TO_LLM \ > $NAME_OR_PATH_TO_ADAPTER \ > $SAVE_PATH \ > --max-shard-size 2GB [2024-02-18 13:09:36,200] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:13<00:00, 1.74s/it] Saving to /root/ex4/hf_merge... All done!
修改web demo:
先安装依赖的软件：
pip install streamlit==1.24.0
屏幕输出：
(ex4) root@intern-studio-069640:~/ex4/data# pip install streamlit==1.24.0 Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting streamlit==1.24.0 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/3d/5f/1716c66c58a8b080f9673e7ca569044b1b8ac6a5cd8519fe4b1afa89abf8/streamlit-1.24.0-py2.py3-none-any.whl (8.9 MB) Collecting altair<6,>=4.0 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c5/e4/7fcceef127badbb0d644d730d992410e4f3799b295c9964a172f92a469c7/altair-5.2.0-py3-none-any.whl (996 kB) Collecting blinker<2,>=1.0.0 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/fa/2a/7f3714cbc6356a0efec525ce7a0613d581072ed6eb53eb7b9754f33db807/blinker-1.7.0-py3-none-any.whl (13 kB) Collecting cachetools<6,>=4.0 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a2/91/2d843adb9fbd911e0da45fbf6f18ca89d07a087c3daa23e955584f90ebf4/cachetools-5.3.2-py3-none-any.whl (9.3 kB) Collecting click<9,>=7.0 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl (97 kB) Collecting importlib-metadata<7,>=1.4 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/59/9b/ecce94952ab5ea74c31dcf9ccf78ccd484eebebef06019bf8cb579ab4519/importlib_metadata-6.11.0-py3-none-any.whl (23 kB) Requirement already satisfied: numpy<2,>=1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (1.26.2) Requirement already satisfied: packaging<24,>=14.1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (23.2) Requirement already satisfied: pandas<3,>=0.25 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (2.2.0) Collecting pillow<10,>=6.2.0 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/25/6b/d3c35d207c9c0b6c2f855420f62e64ef43d348e8c797ad1c32b9f2106a19/Pillow-9.5.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.4 MB) Collecting protobuf<5,>=3.20 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/15/db/7f731524fe0e56c6b2eb57d05b55d3badd80ef7d1f1ed59db191b2fdd8ab/protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl (294 kB) Requirement already satisfied: pyarrow>=4.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (15.0.0) Collecting pympler<2,>=0.9 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/2c/42/41e1469ed0b37b9c8532cb8074bea179f7d85ee7e82a59b5b6c289ed6045/Pympler-1.0.1-py3-none-any.whl (164 kB) Requirement already satisfied: python-dateutil<3,>=2 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (2.8.2) Requirement already satisfied: requests<3,>=2.4 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (2.31.0) Requirement already satisfied: rich<14,>=10.11.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (13.7.0) Collecting tenacity<9,>=8.0.0 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f4/f1/990741d5bb2487d529d20a433210ffa136a367751e454214013b441c4575/tenacity-8.2.3-py3-none-any.whl (24 kB) Collecting toml<2 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl (16 kB) Requirement already satisfied: typing-extensions<5,>=4.0.1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (4.7.1) Collecting tzlocal<5,>=1.1 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/55/a6/a75af44665e5e77e52f6789eef4f8bc056c8e039d96c804b975806942580/tzlocal-4.3.1-py3-none-any.whl (20 kB) Collecting validators<1,>=0.2 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/3a/0c/785d317eea99c3739821718f118c70537639aa43f96bfa1d83a71f68eaf6/validators-0.22.0-py3-none-any.whl (26 kB) Collecting gitpython!=3.1.19,<4,>=3 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/67/c7/995360c87dd74e27539ccbfecddfb58e08f140d849fcd7f35d2ed1a5f80f/GitPython-3.1.42-py3-none-any.whl (195 kB) Collecting pydeck<1,>=0.1.dev5 (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/10/4b/2fc80540e2d3903452245bb657c7f758ec7342420507d1e4091b0161856e/pydeck-0.8.1b0-py2.py3-none-any.whl (4.8 MB) Requirement already satisfied: tornado<7,>=6.0.3 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from streamlit==1.24.0) (6.4) Collecting watchdog (from streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/91/7b/26d2f43aa9fe428416be21ee1cb9ac75638cf302466b7e706c14eeaea42c/watchdog-4.0.0-py3-none-manylinux2014_x86_64.whl (82 kB) Requirement already satisfied: jinja2 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit==1.24.0) (3.1.2) Requirement already satisfied: jsonschema>=3.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit==1.24.0) (4.21.1) Collecting toolz (from altair<6,>=4.0->streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b7/8a/d82202c9f89eab30f9fc05380daae87d617e2ad11571ab23d7c13a29bb54/toolz-0.12.1-py3-none-any.whl (56 kB) Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.19,<4,>=3->streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/fd/5b/8f0c4a5bb9fd491c277c21eff7ccae71b47d43c4446c9d0c6cff2fe8c2c4/gitdb-4.0.11-py3-none-any.whl (62 kB) Requirement already satisfied: zipp>=0.5 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from importlib-metadata<7,>=1.4->streamlit==1.24.0) (3.17.0) Requirement already satisfied: pytz>=2020.1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from pandas<3,>=0.25->streamlit==1.24.0) (2024.1) Requirement already satisfied: tzdata>=2022.7 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from pandas<3,>=0.25->streamlit==1.24.0) (2024.1) Requirement already satisfied: six>=1.5 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from python-dateutil<3,>=2->streamlit==1.24.0) (1.16.0) Requirement already satisfied: charset-normalizer<4,>=2 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests<3,>=2.4->streamlit==1.24.0) (2.0.4) Requirement already satisfied: idna<4,>=2.5 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests<3,>=2.4->streamlit==1.24.0) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests<3,>=2.4->streamlit==1.24.0) (1.26.18) Requirement already satisfied: certifi>=2017.4.17 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from requests<3,>=2.4->streamlit==1.24.0) (2023.11.17) Requirement already satisfied: markdown-it-py>=2.2.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from rich<14,>=10.11.0->streamlit==1.24.0) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from rich<14,>=10.11.0->streamlit==1.24.0) (2.17.2) Collecting pytz-deprecation-shim (from tzlocal<5,>=1.1->streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/eb/73/3eaab547ca809754e67e06871cff0fc962bafd4b604e15f31896a0f94431/pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB) Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.19,<4,>=3->streamlit==1.24.0) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a7/a5/10f97f73544edcdef54409f1d839f6049a0d79df68adbc1ceb24d1aaca42/smmap-5.0.1-py3-none-any.whl (24 kB) Requirement already satisfied: MarkupSafe>=2.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from jinja2->altair<6,>=4.0->streamlit==1.24.0) (2.1.1) Requirement already satisfied: attrs>=22.2.0 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit==1.24.0) (23.2.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit==1.24.0) (2023.12.1) Requirement already satisfied: referencing>=0.28.4 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit==1.24.0) (0.33.0) Requirement already satisfied: rpds-py>=0.7.1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit==1.24.0) (0.18.0) Requirement already satisfied: mdurl~=0.1 in /root/.conda/envs/ex4/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich<14,>=10.11.0->streamlit==1.24.0) (0.1.2) Installing collected packages: watchdog, validators, toolz, toml, tenacity, smmap, pytz-deprecation-shim, pympler, protobuf, pillow, importlib-metadata, click, cachetools, blinker, tzlocal, pydeck, gitdb, gitpython, altair, streamlit Attempting uninstall: pillow Found existing installation: Pillow 10.0.1 Uninstalling Pillow-10.0.1: Successfully uninstalled Pillow-10.0.1 Attempting uninstall: importlib-metadata Found existing installation: importlib-metadata 7.0.1 Uninstalling importlib-metadata-7.0.1: Successfully uninstalled importlib-metadata-7.0.1 Successfully installed altair-5.2.0 blinker-1.7.0 cachetools-5.3.2 click-8.1.7 gitdb-4.0.11 gitpython-3.1.42 importlib-metadata-6.11.0 pillow-9.5.0 protobuf-4.25.3 pydeck-0.8.1b0 pympler-1.0.1 pytz-deprecation-shim-0.1.0.post0 smmap-5.0.1 streamlit-1.24.0 tenacity-8.2.3 toml-0.10.2 toolz-0.12.1 tzlocal-4.3.1 validators-0.22.0 watchdog-4.0.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
从internlm上下载web demo
# 创建code文件夹用于存放InternLM项目代码 mkdir /root/ex4/code && cd /root/ex4/code #git clone https://github.com/InternLM/InternLM.git ###这个地方有问题。最新的代码库上web_demo.py已经移走了。
git clone https://gitee.com/internlm/InternLM.git
cd InternLM
git checkout 3028f07cb79e5b1d7342f4ad8d11efad3fd13d17
屏幕输出：
(ex4) root@intern-studio-069640:~/ex4# mkdir /root/ex4/code && cd /root/ex4/code (ex4) root@intern-studio-069640:~/ex4/code# git clone https://github.com/InternLM/InternLM.git Cloning into 'InternLM'... remote: Enumerating objects: 2878, done. remote: Counting objects: 100% (1609/1609), done. remote: Compressing objects: 100% (599/599), done. remote: Total 2878 (delta 1269), reused 1087 (delta 1002), pack-reused 1269 Receiving objects: 100% (2878/2878), 4.94 MiB | 7.29 MiB/s, done. Resolving deltas: 100% (1841/1841), done. (ex4) root@intern-studio-069640:~/ex4/code#
修改web_demo.py，将模型文件地址修改为微调后的模型文件地址:
最后运行web demo:
streamlit run /root/ex4/code/InternLM/web_demo.py --server.address 127.0.0.1 --server.port 6006
结果：
本地MBP端口映射：
ssh -CNg -L 6006:127.0.0.1:6006 root@ssh.intern-ai.org.cn -p 36970 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
在本地浏览器上打开http://127.0.0.1:6006
posted on 2024-02-18 12:59 littlesuccess 阅读(32) 评论(0) 编辑 收藏 举报
