dremio cloner 简单试用

以前简单介绍过dremio cloner 工具，以下是一个简单试用

dremio 环境准备

基于docker-compose，具体可以参考https://github.com/rongfengliang/dremio_cluster_docker-compose 完成配置

安装dremio cloner
dremio cloner 没有直接提供为一个pip包，需要自己安装
clone 代码

git clone https://github.com/deane-dremio/dremio-cloner.git

安装依赖

cd dremio-cloner.

python -m venv venv

source venv/bin/activate

pip install mo-sql-parsing requests sqlparse

使用

python src/dremio_cloner.py config_dr_read.json

config_dr_read.json 是从config 中拷贝的，可以结合自己的实际选择进行修改，比如我修改的

{"dremio_cloner": [

  {"command":"get"},

  {"source": [

    {"endpoint": "http://localhost:9047/"},

    {"username": "<username>"},

    {"password": "<password>"}]

  },

  {"target": [

    {"filename":"myapp.json"},

    {"overwrite": "False"}]

  },

  {"options": [

    {"logging.level":"logging.ERROR"},

    {"logging.format":"%(levelname)s:%(asctime)s:%(message)s"},

    {"logging.filename":"message.log"},

    {"logging.verbose": "False"},
 
    {"max_errors":"9999"},

    {"http_timeout":"10"},
 
    {"user.process_mode":"skip"},

    {"group.process_mode":"skip"},

    {"space.process_mode":"skip"},

    {"source.process_mode":"process"},

    {"reflection.process_mode": "skip"},

    {"folder.process_mode": "skip"},

    {"vds.process_mode": "skip"},

    {"pds.process_mode": "process"},

    {"wlm.queue.process_mode": "skip"},

    {"wlm.rule.process_mode": "skip"},

    {"wiki.process_mode": "skip"},

    {"tag.process_mode": "skip"},

    {"home.process_mode": "skip"},

    {"vote.process_mode": "skip"},
 
    {"space.filter": "*"},

    {"space.exclude.filter": ""},

    {"space.folder.filter":"*"},

    {"space.folder.exclude.filter":""},
 
    {"source.filter":"*"},

    {"source.exclude.filter":""},

    {"source.folder.filter":"*"},

    {"source.folder.exclude.filter":""},
 
    {"pds.filter":"*"},

    {"pds.exclude.filter":""},

    {"pds.list.useapi":"False"},
 
    {"vds.filter":"*"},

    {"vds.exclude.filter":""},

    {"vds.dependencies.process_mode":"ignore"},
 
      {"reflection.only_for_matching_vds":"True"}]
 
  }]

}