torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use).

解决方案是在torchrun中添加参数--master_port改变master port。且注意这个参数一定要加在要跑的文件即src/entry_point/train.py之前，否则会被忽略。

引用：https://juejin.cn/post/7260668104752775228

我的代码是：
torchrun --nproc_per_node 1 --master_port 29501 -m training.run --output_dir test_path --model_name_or_path openaccess-ai-collective/tiny-mistral --train_data training/toy_data/toy_data_embedding.jsonl --learning_rate 1e-5 --num_train_epochs 5 --per_device_train_batch_size 2 --dataloader_drop_last True --normalized True --temperature 0.02 --query_max_len 32 --passage_max_len 128 --train_group_size 2 --mode embedding --attn cccc