doris docker 部署 及监控 (更新到1.0)

evn:  ubuntu20.04  jdk8_172  doris 0.13.12

 

############# 1 启动Fe  BE  ###################

   准备：

sudo mkdir  -p  /opt/docker/doris/ && sudo chmod -R 777 /opt/docker/doris/

 

PRIORITY_NETWORKS 可以通过 > ip a 查看
运行(启动fe ):  
 sudo docker run -itd --privileged=true --restart=always --name doris-fe_1_0  --net=host  -e FE_ROLE=fe-leader  -e PRIORITY_NETWORKS=192.168.18.176/24 \
-v /opt/docker/doris/fe/log:/data/fe/log  -v /opt/docker/doris/fe/doris-meta:/data/fe/doris-meta  -v /etc/localtime:/etc/localtime:ro  \
   -v /etc/timezone:/etc/timezone    happysea/doris:1.0
运行(启动be ): 
sudo docker  run -itd --privileged=true  --restart=always  --name doris-be_1_0   --net=host -e FE_ROLE=be  --ulimit nofile=102400:102400 \
-e PRIORITY_NETWORKS=192.168.18.54/24  \
-v /opt/docker/doris/be/log:/data/be/log   -v /opt/docker/doris/be/storage:/data/be/storage  -v /etc/localtime:/etc/localtime:ro   \
 -v /etc/timezone:/etc/timezone   happysea/doris:1.0


 关于文件数: 可以添加


--ulimit nofile=102400:102400 
like:


sudo docker  run -itd --privileged=true --name doris-be --net=host --restart=always  --ulimit nofile=102400:102400  -e xx......   images


 


JVM　参数是默认最大4G ,建议改为８Ｇ＞如果需要：


可以修改：


在启动添加参数： -e  JAVA_OPTS="-Xmx4096m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"


############# 2.连接添加Be  ###################


连接fe
mysql -h host -P 9030 -uroot


修改root 用户密码 (如果需要 ):   >   set password for 用户名@localhost = password('新密码');  


其中 host 为 FE 所在节点 ip；port 为 fe/conf/fe.conf 中的 query_port；默认使用 root 账户，无密码登录。 


 


添加be
ALTER SYSTEM ADD BACKEND “host1:9050”;
ALTER SYSTEM ADD BACKEND “host2:9050”;
ALTER SYSTEM ADD BACKEND “host3:9050”;
(
如果使用多组户功能，则执行以下命令添加BE:
ALTER SYSTEM ADD FREE BACKEND “host:port”;
)


其中 host 为 BE 所在节点 ip；port 为 be/conf/be.conf 中的 heartbeat_service_port。


    

  • 

    BE 进程将启动并进入后台执行。日志默认存放在 be/log/ 目录下。如启动失败，可以通过查看 be/log/be.log 或者 be/log/be.out 查看错误信息。
    






    • 

  • 

    查看BE状态
    

    使用 mysql-client 连接到 FE，并执行 SHOW PROC '/backends'; 查看 BE 运行情况。如一切正常，isAlive 列应为 true。 
    





    • 







 删除be:  ALTER SYSTEM DECOMMISSION BACKEND "192.168.18.52:9050";


 


查看 fe与be状态
show proc "/frontends";
show proc "/backends";


 


 


 


 


 


 


 


 


 


 


 


 


 


 


 


可以直接下载编译好的: 


 http://doris.incubator.apache.org/zh-CN/downloads/downloads.html#apache-doris


 


Dockerfile:




FROM happysea/jdk8
MAINTAINER Sea <lshan523@163.com>
RUN mkdir -p /data/be/log/
RUN mkdir -p /data/be/storage
RUN mkdir -p /data/fe/doris-meta
RUN mkdir -p /data/fe/log
RUN chmod -R  755  /data/
COPY doris/ /opt/doris/
COPY docker-entrypoint.sh /opt/
WORKDIR  /opt
EXPOSE 8030 9020 9030 9060 9070 8040 9050 8060 
#ENTRYPOINT["/bin/bash"]
ENTRYPOINT ["/bin/bash","/opt/docker-entrypoint.sh"]




 


docker-entrypoint.sh




#!/bin/bash
chmod  -R  755 /data/
echo "fe_role:"$FE_ROLE
echo "leader:"$FE_LEADER

source /etc/profile

if [[ $FE_ROLE = "fe-leader" ]]; then
    echo "fe-leader"
    /opt/doris/fe/bin/start_fe.sh
elif [[ $FE_ROLE = "be" ]]; then
      echo "be"
     /opt/doris/be/bin/start_be.sh
elif [[ $FE_ROLE = "fe-follower" ]]; then
    /opt/doris/fe/bin/start_fe.sh --helper $FE_LEADER
else
    #/opt/doris/fs_broker/bin/start_broker.sh
    echo "do nothing "
fi




打包: sudo docker build -t  happysea/doris:latest


 


 


 


docker-compose 参考如下即可:


 


以下可以参考：


Docker搭建Doris集群监控grafana+prometheus： https://blog.csdn.net/zuoliansheng/article/details/108736188


使用Docker快速搭建Doris集群： https://blog.csdn.net/jklcl/article/details/112910796


》


制作docker镜像




# centos7:jdk8是我们自己做的基础镜像
FROM centos7:jdk8

RUN mkdir -p /home/doris

ENV JAVA_HOME /usr/lib/jvm/java

COPY ./fe/ /home/doris/fe

COPY ./be/ /home/doris/be

COPY ./apache_hdfs_broker/ /home/doris/fs_broker

EXPOSE 8030 9020 9030 9010 9070 9060 8060 8040 9050 8000

VOLUME ["/home/doris/fe/conf", "/home/doris/fe/log", "/home/doris/fe/doris-meta", "/home/doris/be/conf", "/home/doris/be/log", "/home/doris/be/storage", "/home/doris/fs_brokers/conf"]

COPY entrypoint.sh /

RUN chmod +x entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]




 


entrypoint.sh




#!/bin/sh

echo "fe_role:"$FE_ROLE
echo "leader:"$FE_LEADER

if [[ $FE_ROLE = 'fe-leader' ]]; then
    /home/doris/fe/bin/start_fe.sh
elif [[ $FE_ROLE = 'be' ]]; then
    /home/doris/be/bin/start_be.sh
elif [[ $FE_ROLE = 'fe-follower' ]]; then
    /home/doris/fe/bin/start_fe.sh --helper $FE_LEADER
else
    /home/doris/fs_broker/bin/start_broker.sh
fi




 


创建镜像




docker build -f DockerFile -t doris:0.12.21-release .




docker-compose启动


FE




version: '3.7'
services:
    doris-fe:
        image: doris:0.12.21-release
        restart: always
        network_mode: "host"
        container_name: "doris-fe"
        ports:
            - "8030:8030"
            - "9010:9010"
            - "9020:9020"
            - "9030:9030"
        volumes:
            - "/xxx/doris/fe/log:/home/doris/fe/log"
            - "/xxx/doris/fe/doris-meta:/home/doris/fe/doris-meta"
            - "/xxx/doris/fe/conf:/home/doris/fe/conf"
            - "/etc/localtime:/etc/localtime:ro"
        environment:
            - FE_ROLE=fe-follower
            - FE_LEADER=xxxx:9010
        security_opt:
            - seccomp:unconfined




创建文件夹: mkdir /xxx/doris/fe/conf
启动leader: docker-compose -f docker-compose-doris-fe-leader.yml up -d
启动follower: 略
UI界面: http://s-hadoop-log01:8030/
注：第一次启动时先启动leader，再启动follower，因为follower指向leader所在机器之后的启动，都按照leader起就可以了



BE




version: '3.7'
services:
    doris-be:
        image: doris:0.12.21-release
        restart: always
        network_mode: "host"
        container_name: "doris-be"
        ports:
            - "8040:8040"
            - "8060:8060"
            - "9050:9050"
            - "9060:9060"
            - "9070:9070"
        volumes:
            - "/xxx/doris/be/log:/home/doris/be/log"
            - "/xxx/doris/be/storage:/home/doris/be/storage"
            - "/xxx/doris/be/conf:/home/doris/be/conf/"
            - "/etc/localtime:/etc/localtime:ro"
        environment:
            - FE_ROLE=be




创建文件夹: mkdir /xxx/doris/be/conf
启动be: docker-compose -f docker-compose-doris-be.yml up -d
UI界面: http://s-hadoop-log01:8040/


 


doris-fs-broker:




version: '3.7'
services:
    doris-fs-broker:
        image: doris:0.12.21-release
        restart: always
        network_mode: "host"
        container_name: "doris-fs-broker"
        ports:
            - "8000:8000"
        volumes:
            - "/xxx/doris/fs_broker/conf:/home/doris/fs_broker/conf"
            - "/xxx/doris/fs_broker/log:/home/doris/fs_broker/log"
            - "/etc/localtime:/etc/localtime:ro"
        environment:
            - FE_ROLE=fs




创建文件夹: mkdir /xxx/doris/fs_broker/conf
启动fs: docker-compose -f docker-compose-doris-fs-broker.yml up -d


 


 






        
posted on 
2021-05-19 10:27 
lshan 
阅读(1780) 
评论(0) 
编辑 
收藏 
举报


    










    
    

    
    

    

        

    

        

            
                
                
            
        

    

    

    

    





	


	Copyright © 2024 lshan

Powered by .NET 8.0 on Kubernetes
 Powered by: 博客园