druid安装部署

Druid安装部署

一、环境需求

Java 8 (8u92 +)

Linux, Mac OS X, or other Unix-like OS(不支持Windows系统)

Zookeeper (3.4 +)

Druid下载:

官网:https://druid.apache.org,进入官网后点击“Download”,进入下载界面,当前最新版本0.16,这里我们选择编译版本下载;

如果想下载历史版本,可以下拉,点击“Apache release archives “进入历史版本下载界面

 

二、单机版安装部署

1.将安装包上传到服务器,并解压安装包

tar -xzf apache-druid-0.16.0-incubating-bin.tar.gz  -C  /data

2.下载zookeeper安装包上传到服务器,并解压安装包到druid的根目录,并重名为zk

  tar -xzf zookeeper-3.4.6.tar.gz  -C /data/apache-druid-0.16.0-incubating

  mv  /data/apache-druid-0.16.0-incubating/zookeeper-3.4.6  /data /apache-druid-0.16.0-incubating/zk

3.进入druid的安装目录,执行单机启动脚本

  cd /data/apache-druid-0.16.0-incubating

  ./bin/start-micro-quickstart

      4.访问http://localhost:8888 查看druid管理界面

三、集群版安装部署

部署规划

角色

机器

集群角色(进程)

主节点

hadoop1(192.168.252.111)

Coordinator,Overlord

数据节点

hadoop2(192.168.252.112)

Historical, MiddleManager

查询节点

hadoop3(192.168.252.113)

Broker ,router

外部依赖

组件

机器

描述

zookeeper

hadoop1、hadoop2、hadoop3

提供分布式协调服务

hadoop

hadoop1、hadoop2、hadoop3

提供数据文件存储库

mysql

hadoop1

提供元数据存储库

注:外部依赖组件安装请查阅相关资料,这里默认外部依赖已安装成功并运行

1.将安装包上传到hadoop1服务器,并解压安装包,并进入该目录

tar -xzf apache-druid-0.16.0-incubating-bin.tar.gz  -C  /data

cd /data/apache-druid-0.16.0-incubating

2.修改核心配置文件,配置文件如下

      vim conf/druid/cluster/_common/common.runtime.properties

  1.  
    #
  2.  
    # Licensed to the Apache Software Foundation (ASF) under one
  3.  
    # or more contributor license agreements. See the NOTICE file
  4.  
    # distributed with this work for additional information
  5.  
    # regarding copyright ownership. The ASF licenses this file
  6.  
    # to you under the Apache License, Version 2.0 (the
  7.  
    # "License"); you may not use this file except in compliance
  8.  
    # with the License. You may obtain a copy of the License at
  9.  
    #
  10.  
    # http://www.apache.org/licenses/LICENSE-2.0
  11.  
    #
  12.  
    # Unless required by applicable law or agreed to in writing,
  13.  
    # software distributed under the License is distributed on an
  14.  
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  15.  
    # KIND, either express or implied. See the License for the
  16.  
    # specific language governing permissions and limitations
  17.  
    # under the License.
  18.  
    #
  19.  
     
  20.  
    # Extensions specified in the load list will be loaded by Druid
  21.  
    # We are using local fs for deep storage - not recommended for production - use S3, HDFS, or NFS instead
  22.  
    # We are using local derby for the metadata store - not recommended for production - use MySQL or Postgres instead
  23.  
     
  24.  
    # If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system.
  25.  
    # If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory.
  26.  
    # More info: https://druid.apache.org/docs/latest/operations/including-extensions.html
  27.  
    druid.extensions.loadList=["mysql-metadata-storage","druid-hdfs-storage","druid-kafka-indexing-service"]
  28.  
     
  29.  
    # If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
  30.  
    # and uncomment the line below to point to your directory.
  31.  
    #druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
  32.  
     
  33.  
     
  34.  
    #
  35.  
    # Hostname
  36.  
    #
  37.  
    druid.host=localhost
  38.  
     
  39.  
    #
  40.  
    # Logging
  41.  
    #
  42.  
     
  43.  
    # Log all runtime properties on startup. Disable to avoid logging properties on startup:
  44.  
    druid.startup.logging.logProperties=true
  45.  
     
  46.  
    #
  47.  
    # Zookeeper
  48.  
    #
  49.  
     
  50.  
    druid.zk.service.host=hadoop1:2181,hadoop2:2181,hadoop3:2181
  51.  
    druid.zk.paths.base=/druid
  52.  
     
  53.  
    #
  54.  
    # Metadata storage
  55.  
    #
  56.  
     
  57.  
    # For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
  58.  
    #druid.metadata.storage.type=derby
  59.  
    #druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
  60.  
    #druid.metadata.storage.connector.host=localhost
  61.  
    #druid.metadata.storage.connector.port=1527
  62.  
     
  63.  
    # For MySQL (make sure to include the MySQL JDBC driver on the classpath):
  64.  
    druid.metadata.storage.type=mysql
  65.  
    druid.metadata.storage.connector.connectURI=jdbc:mysql://hadoop1:3306/druid
  66.  
    druid.metadata.storage.connector.user=root
  67.  
    druid.metadata.storage.connector.password=123456
  68.  
     
  69.  
    # For PostgreSQL:
  70.  
    #druid.metadata.storage.type=postgresql
  71.  
    #druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid
  72.  
    #druid.metadata.storage.connector.user=...
  73.  
    #druid.metadata.storage.connector.password=...
  74.  
     
  75.  
    #
  76.  
    # Deep storage
  77.  
    #
  78.  
     
  79.  
    # For local disk (only viable in a cluster if this is a network mount):
  80.  
    #druid.storage.type=local
  81.  
    #druid.storage.storageDirectory=var/druid/segments
  82.  
     
  83.  
    # For HDFS:
  84.  
    druid.storage.type=hdfs
  85.  
    druid.storage.storageDirectory=hdfs://testcluster/druid/segments
  86.  
     
  87.  
    # For S3:
  88.  
    #druid.storage.type=s3
  89.  
    #druid.storage.bucket=your-bucket
  90.  
    #druid.storage.baseKey=druid/segments
  91.  
    #druid.s3.accessKey=...
  92.  
    #druid.s3.secretKey=...
  93.  
     
  94.  
    #
  95.  
    # Indexing service logs
  96.  
    #
  97.  
     
  98.  
    # For local disk (only viable in a cluster if this is a network mount):
  99.  
    #druid.indexer.logs.type=file
  100.  
    #druid.indexer.logs.directory=var/druid/indexing-logs
  101.  
     
  102.  
    # For HDFS:
  103.  
    druid.indexer.logs.type=hdfs
  104.  
    druid.indexer.logs.directory=hdfs://testcluster/druid/indexing-logs
  105.  
     
  106.  
    # For S3:
  107.  
    #druid.indexer.logs.type=s3
  108.  
    #druid.indexer.logs.s3Bucket=your-bucket
  109.  
    #druid.indexer.logs.s3Prefix=druid/indexing-logs
  110.  
     
  111.  
    #
  112.  
    # Service discovery
  113.  
    #
  114.  
     
  115.  
    druid.selectors.indexing.serviceName=druid/overlord
  116.  
    druid.selectors.coordinator.serviceName=druid/coordinator
  117.  
     
  118.  
    #
  119.  
    # Monitoring
  120.  
    #
  121.  
     
  122.  
    druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
  123.  
    druid.emitter=noop
  124.  
    druid.emitter.logging.logLevel=info
  125.  
     
  126.  
    # Storage type of double columns
  127.  
    # ommiting this will lead to index double as float at the storage layer
  128.  
     
  129.  
    druid.indexing.doubleStorage=double
  130.  
     
  131.  
    #
  132.  
    # Security
  133.  
    #
  134.  
    druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
  135.  
     
  136.  
     
  137.  
    #
  138.  
    # SQL
  139.  
    #
  140.  
    druid.sql.enable=true
  141.  
     
  142.  
    #
  143.  
    # Lookups
  144.  
    #
  145.  
    druid.lookup.enableLookupSyncOnStartup=false

重点关注红色部分,其中druid.extensions.loadList为需要加载的外部组件配置,示例中依赖外部组件zookeeper、hadoop、mysql。(kafka依赖配置可选)

zookeeper默认配置支持zookeeper3.4.14,如何外部依赖zookeeper版本不兼容,则需要将相关的zookeeper jar拷贝到druid的lib目录下替换原zookeeper相关jar

hadoop 默认配置支持hadoop 2.8.3客户端,该处需要将外部依赖中hadoop的核心配置文件core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml拷贝到druid中的conf/druid/cluster/_common/目录下,如果该hadoop客户端不兼容组件中的hadoop集群,还需要将集群兼容的hadoop客户端相关jar拷贝到druid中的extensions/druid-hdfs-storage目录下替换原来相关jar

mysql 没有默认支持,需要将相关版本的mysql的驱动jar拷贝到extensions/mysql-metadata-storage目录下

3.将druid整个目录发送到hadoop2和hadoop3机器中

4.根据需要(机器配置、性能要求等)修改hadoop1中关于master节点相关配置,在Druid中的conf\druid\cluster\master\coordinator-overlord目录下,其中runtime.properties为druid想的属性配置,jvm.config为该节点进程的jvm参数配置,默认的集群版本的jvm配置均比较高,如果用于学习测试机器配置不够用情况下,可以将单机版的配置拷贝过来,单机版配置目录conf\druid\single-server\micro-quickstart\coordinator-overlord

5.根据需要(机器配置、性能要求等)修改hadoop2中关于data节点相关配置,在Druid中的conf\druid\cluster\ data目录下,该节点有两个进程historical和middleManager,根据需要修改相关配置,类似步骤4如果测试可以选择单机版的相关配置

6. 根据需要(机器配置、性能要求等)修改hadoop3中关于query节点相关配置,在Druid中的conf\druid\cluster\ query目录下,该节点有两个进程broker和router,根据需要修改相关配置,类似步骤4如果测试可以选择单机版的相关配置

注:这里druid集群版本中默认配置较测试学习虚拟机要求过高,笔者测试均采用单机版配置

7.启动

      hadoop1:sh bin/start-cluster-master-no-zk-server >master.log &

      hadoop2:sh bin/start-cluster-data-server > data.log &

      hadoop3:sh bin/start-cluster-query-server > query.log &

8.关闭:只需要在相关的节点执行 bin/service  --down 关闭当前节点的druid相关的全部进程

9.Druid相关管理界面访问地址:

      http://192.168.252.113:8888/

      http://192.168.252.111:8081/

 

常见问题

1.关闭命令,不能使用sh bin/service  --down ,因为service  脚本获取参数缺陷

2. bin/verify-default-ports 脚本为启动检查端口占用脚本,如果你修改过相关组件的端口后,还是提示端口占用,需要关注该脚本,同步修改(该情况一般出现在单机版中)

3.如果启动日志中出现:Unrecogized VM option 'ExitOnOutOfMemoryError' 说明内存不足,需要修改相应进程的jvm.config配置,或者加机器内存

https://blog.csdn.net/cb2474600377/article/details/103577796

posted @ 2020-11-20 15:59  大浪不惊涛  阅读(2816)  评论(0编辑  收藏  举报