milvus-migration安装使用

环境:
OS:Centos 7
源端和目的端milvus版本:2.4.6
源端ip:192.168.1.134
目的端ip:192.168.1.135

1.下载地址
https://github.com/zilliztech/milvus-migration/releases/tag/v0.1.26

2.安装
[root@host135 opt]# cd /opt
[root@host135 opt]# mkdir milvus-migration
[root@host135 opt]# mv milvus-migration_Linux_x86_64.tar.gz ./milvus-migration/
[root@host135 milvus-migration]# tar -xvf milvus-migration_Linux_x86_64.tar.gz

解压后文件如下:

[root@host135 milvus-migration]# ls -al
total 70296
drwxrwxr-x   2 root   root        195 Nov 13 09:35 .
drwxr-xr-x. 37 root   root       4096 Nov 13 09:34 ..
-rwxr-xr-x   1 oracle docker 55238808 Aug 23 20:00 milvus-migration
-rw-rw-r--   1 root   root   16671683 Nov 13 09:32 milvus-migration_Linux_x86_64.tar.gz
-rw-r--r--   1 oracle docker     9793 Aug 23 19:58 README_1X.md
-rw-r--r--   1 oracle docker     5454 Aug 23 19:58 README_2X.md
-rw-r--r--   1 oracle docker    12359 Aug 23 19:58 README_ES.md
-rw-r--r--   1 oracle docker     7565 Aug 23 19:58 README_FAISS.md
-rw-r--r--   1 oracle docker     3543 Aug 23 19:58 README.md
-rw-r--r--   1 oracle docker    10790 Aug 23 19:58 README_old.md

若我们这里是milvus 2迁移到milvus 2的话,阅读README_2X.md

 

3.从README_2X.md获取配置文件内容

dumper:
  worker:
    workMode: milvus2x      # work mode：milvus2x->milvus2x
    reader:
      bufferSize: 500       # Read source data rows in each time read from Source Milvus.

meta:                       # meta part
  mode: config              # 'config' mode means will get meta config from this config file itself.
  version: 2.3.0            #  Source Milvus version
  collection: src_coll_name # migrate data from this source collection

source:                     # source milvus connection info
  milvus2x:
    endpoint: {milvus2x_domain}:{milvus2x_port}
    username: xxxx
    password: xxxxx

target:                    # target milvus collection info
  milvus2x:
    endpoint: {milvus2x_domain}:{milvus2x_port}
    username: xxxx
    password: xxxxx

 

将如上内容保存到 migration.yaml,我们这里创建配置文件目录

[root@host135 milvus-migration]#cd /opt/milvus-migration
[root@host135 milvus-migration]#mkdir configs

 

最后的配置文件内容如下(我这里是迁移db_test库下的表car):

[root@host135 configs]# more migration.yaml 
dumper:
  worker:
    workMode: milvus2x      # work mode：milvus2x->milvus2x
    reader:
      bufferSize: 500       # Read source data rows in each time read from Source Milvus.

meta:                       # meta part
  mode: config              # 'config' mode means will get meta config from this config file itself.
  version: 2.4.6            #  Source Milvus version
  collection: car # migrate data from this source collection

source:                     # source milvus connection info
  milvus2x:
    endpoint: 192.168.1.134:19530
    database: db_test
    username: root
    password: Milvus

target:                    # target milvus collection info
  milvus2x:
    endpoint: 192.168.1.135:19530
    database: db_test
    username: root
    password: Milvus

 

4.启动
[root@host135 milvus-migration]# cd /opt/milvus-migration
[root@host135 milvus-migration]#./milvus-migration start

若配置文件不在configs目录,可以在命令里面指定配置文件
[root@host135 milvus-migration]#./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml

或者在命令行通过-t参数指定某个collection

[root@host135 milvus-migration]#./milvus-migration start -t="tb_test02" -c=/opt/milvus-migration/configs/migration.yaml
这里指定了-t参数的话,配置文件里指定的collection: car参数就不管用了.

 

5.同步某个库多个表
脚本如下:
vi milvus_migration_all_collection.sh

#!/bin/bash

collections=("car" "tb_test02" "CEC_Corpus")

for collection in "${collections[@]}"; do
echo "BatchMigration==> $collection"
/opt/milvus-migration/milvus-migration start -t="$collection" -c=/opt/milvus-migration/configs/migration.yaml
done

 

若是需要迁移的表很多的情况下可以使用如下方式：

mapfile -t collections < /tmp/input.txt 

把需要迁移的表放到input.txt文件，每个表名一行:

car

tb_test02

CEC_Corpus

 

 

注意事项:
1.源端的collection必须要load,否则包如下的错误
error="failed to query: collection not loaded

2.目的端不需要提前创建好数据库,同步后会自动创建数据库

3.同步后的collection是没有索引的,当然也没有load,需要手工创建索引;