Hadoop生态圈-CentOs7.5单机部署ClickHouse
Hadoop生态圈-CentOs7.5单机部署ClickHouse
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
到了新的公司，认识了新的同事，生产环境也得你去适应新的集群环境，我新入职的公司的大数据开发同时并不看好hbase，而是对clickhouse青睐有加，听他们口头把ClickHouse吹的那么的出神入化，听的我是有种迫不及待想要对它一探究竟。
最新的一手资料请参考官网：https://clickhouse.yandex/
一.ClickHouse的独特功能
1>.真正的面向列的DBMS
因为有些系统可以单独存储不同列的值，但由于它们针对其他场景进行了优化，因此无法有效地处理分析查询。例如HBase，BigTable，Cassandra和HyperTable。在这些系统中，您将获得大约每秒十万行的吞吐量，但不会达到每秒数亿行。
值得注意的是，ClickHouse是一个数据库管理系统，而不是一个数据库。ClickHouse允许在运行时创建表和数据库，加载数据和运行查询，而无需重新配置和重新启动服务器。
2>.适用于在线查询
低延迟意味着可以在没有延迟的情况下处理查询，而无需在加载用户界面页面的同一时间提前准备答案。换句话说，在线。
3>.数据复制和数据完整性支持
使用异步多主机复制。写入任何可用副本后，数据将分发到后台的所有剩余副本。系统在不同的副本上维护相同的数据。在大多数故障后自动执行恢复，在复杂情况下 - 半自动执行。
4>.支持近似计算
ClickHouse提供了各种交易性能准确度的方法：
4.1>.聚合函数，用于近似计算不同值，中位数和分位数的数量。
4.2>.基于数据的一部分（样本）运行查询并获得近似结果。在这种情况下，从磁盘中检索的数据比例较少。
4.2>.为有限数量的随机密钥运行聚合，而不是为所有密钥运行聚合。在数据中密钥分发的某些条件下，这在使用较少资源的同时提供了相当准确的结果。
5>.SQL支持
ClickHouse支持基于SQL的声明性查询语言，在许多情况下与SQL标准相同。支持的查询包括GROUP BY，ORDER BY，FROM，IN和JOIN子句中的子查询以及标量子查询。不支持从属子查询和窗口函数。
以上说明并非我个人对ClickHouse的夸张说法，而是摘自官方文档的说明，可能官网对其软件有一定的夸张手法在里面，官网文档请参考：https://clickhouse.yandex/docs/en/
二.部署ClickHouse
推荐的安装方法参考：https://github.com/Altinity/clickhouse-rpm-install。文档写的头头是道，说下载一个安装shell脚本，结果你按照它的方法做会跑出一系列的坑，我在这里就不吐槽了，喜欢被虐的小伙伴可以去试试，我生产环境用的操作系统基本上都是CentOs的操作系统，我参考了网上多位网友的部署方法，找到了一种最简单有效的部署方法，我这里一句把坑给大家踩过了，只要我的按照步骤来部署就能成功！
1>.操作平台介绍
[root@yinzhengjie ~]# ip a | grep global | awk '{print $2}' | awk -F '/' '{print $1}' 10.1.2.105 [root@yinzhengjie ~]# [root@yinzhengjie ~]# hostname yinzhengjie [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo| grep "processor"| wc -l 14 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo| grep "cpu cores"| uniq cpu cores : 7 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l 2 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c 14 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz [root@yinzhengjie ~]# [root@yinzhengjie ~]# free -m total used free shared buff/cache available Mem: 11852 1406 8309 103 2136 9954 Swap: 6015 0 6015 [root@yinzhengjie ~]# [root@yinzhengjie ~]# free -g total used free shared buff/cache available Mem: 11 1 8 0 2 9 Swap: 5 0 5 [root@yinzhengjie ~]# [root@yinzhengjie ~]# [root@yinzhengjie ~]# uname -r 3.10.0-862.el7.x86_64 [root@yinzhengjie ~]# [root@yinzhengjie ~]# uname -m x86_64 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) You have new mail in /var/spool/mail/root [root@yinzhengjie ~]#
2>.检查是否支持SSE4.2
[root@yinzhengjie ~]# grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported" SSE 4.2 supported [root@yinzhengjie ~]#
3>.下载依赖包
我已经将上面的rpm包下载好了，以及需要的依赖包都下载好啦～大家可以直接下载来用即可。百度云链接:https://pan.baidu.com/s/16gPGxOnHKfRZj2ytOil7qQ 密码:rs4u
[root@yinzhengjie download]# ll total 42916 -rw-r--r-- 1 root root 43945614 Oct 10 20:10 yinzhengjie-clickhouse.tar.gz [root@yinzhengjie download]# [root@yinzhengjie download]# [root@yinzhengjie download]# tar zxf yinzhengjie-clickhouse.tar.gz -C ./ [root@yinzhengjie download]# [root@yinzhengjie download]# ll total 42920 drwxr-xr-x 2 root root 4096 Oct 10 20:03 clickhouse -rw-r--r-- 1 root root 43945614 Oct 10 20:10 yinzhengjie-clickhouse.tar.gz [root@yinzhengjie download]# [root@yinzhengjie download]# cd clickhouse/ [root@yinzhengjie clickhouse]# [root@yinzhengjie clickhouse]# ll total 43080 -rw-r--r-- 1 root root 3048 Oct 10 17:16 clickhouse-client-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 901472 Oct 10 17:16 clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 9043256 Oct 10 17:16 clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 33680608 Oct 10 17:18 clickhouse-server-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 8164 Oct 10 17:16 clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 49036 Oct 10 18:35 libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm -rw-r--r-- 1 root root 421792 Oct 10 18:20 unixODBC-2.3.1-11.el7.x86_64.rpm [root@yinzhengjie clickhouse]#
4>.安装ClickHouse的依赖及其rpm包
[root@yinzhengjie clickhouse]# ll total 43080 -rw-r--r-- 1 root root 3048 Oct 10 17:16 clickhouse-client-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 901472 Oct 10 17:16 clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 9043256 Oct 10 17:16 clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 33680608 Oct 10 17:18 clickhouse-server-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 8164 Oct 10 17:16 clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 49036 Oct 10 18:35 libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm -rw-r--r-- 1 root root 421792 Oct 10 18:20 unixODBC-2.3.1-11.el7.x86_64.rpm [root@yinzhengjie clickhouse]# [root@yinzhengjie clickhouse]# You have new mail in /var/spool/mail/root [root@yinzhengjie clickhouse]# [root@yinzhengjie clickhouse]# yum -y localinstall *.rpm Loaded plugins: fastestmirror Examining clickhouse-client-1.1.54236-4.el7.x86_64.rpm: clickhouse-client-1.1.54236-4.el7.x86_64 Marking clickhouse-client-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm: clickhouse-compressor-1.1.54236-4.el7.x86_64 Marking clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm: clickhouse-debuginfo-1.1.54236-4.el7.x86_64 Marking clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-server-1.1.54236-4.el7.x86_64.rpm: clickhouse-server-1.1.54236-4.el7.x86_64 Marking clickhouse-server-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm: clickhouse-server-common-1.1.54236-4.el7.x86_64 Marking clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm to be installed Examining libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm: libtool-ltdl-2.4.2-21.el7_2.x86_64 Marking libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm to be installed Examining unixODBC-2.3.1-11.el7.x86_64.rpm: unixODBC-2.3.1-11.el7.x86_64 Marking unixODBC-2.3.1-11.el7.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package clickhouse-client.x86_64 0:1.1.54236-4.el7 will be installed ---> Package clickhouse-compressor.x86_64 0:1.1.54236-4.el7 will be installed --> Processing Dependency: libicudata.so.50()(64bit) for package: clickhouse-compressor-1.1.54236-4.el7.x86_64 Determining fastest mirrors 10gen | 2.5 kB 00:00:00 base | 3.6 kB 00:00:00 centosplus | 3.4 kB 00:00:00 epel | 3.2 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/3): epel/x86_64/updateinfo | 948 kB 00:00:00 (2/3): epel/x86_64/primary | 3.6 MB 00:00:01 (3/3): updates/7/x86_64/primary_db | 6.0 MB 00:00:01 epel 12723/12723 --> Processing Dependency: libicui18n.so.50()(64bit) for package: clickhouse-compressor-1.1.54236-4.el7.x86_64 --> Processing Dependency: libicuuc.so.50()(64bit) for package: clickhouse-compressor-1.1.54236-4.el7.x86_64 ---> Package clickhouse-debuginfo.x86_64 0:1.1.54236-4.el7 will be installed ---> Package clickhouse-server.x86_64 0:1.1.54236-4.el7 will be installed ---> Package clickhouse-server-common.x86_64 0:1.1.54236-4.el7 will be installed ---> Package libtool-ltdl.x86_64 0:2.4.2-21.el7_2 will be installed ---> Package unixODBC.x86_64 0:2.3.1-11.el7 will be installed --> Running transaction check ---> Package libicu.x86_64 0:50.1.2-15.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: clickhouse-client x86_64 1.1.54236-4.el7 /clickhouse-client-1.1.54236-4.el7.x86_64 110 clickhouse-compressor x86_64 1.1.54236-4.el7 /clickhouse-compressor-1.1.54236-4.el7.x86_64 2.9 M clickhouse-debuginfo x86_64 1.1.54236-4.el7 /clickhouse-debuginfo-1.1.54236-4.el7.x86_64 48 M clickhouse-server x86_64 1.1.54236-4.el7 /clickhouse-server-1.1.54236-4.el7.x86_64 133 M clickhouse-server-common x86_64 1.1.54236-4.el7 /clickhouse-server-common-1.1.54236-4.el7.x86_64 16 k libtool-ltdl x86_64 2.4.2-21.el7_2 /libtool-ltdl-2.4.2-21.el7_2.x86_64 66 k unixODBC x86_64 2.3.1-11.el7 /unixODBC-2.3.1-11.el7.x86_64 1.2 M Installing for dependencies: libicu x86_64 50.1.2-15.el7 base 6.9 M Transaction Summary =================================================================================================================================================================================================================== Install 7 Packages (+1 Dependent package) Total size: 192 M Total download size: 6.9 M Installed size: 209 M Downloading packages: libicu-50.1.2-15.el7.x86_64.rpm | 6.9 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libtool-ltdl-2.4.2-21.el7_2.x86_64 1/8 Installing : unixODBC-2.3.1-11.el7.x86_64 2/8 Installing : clickhouse-server-common-1.1.54236-4.el7.x86_64 3/8 Installing : libicu-50.1.2-15.el7.x86_64 4/8 Installing : clickhouse-server-1.1.54236-4.el7.x86_64 5/8 Installing : clickhouse-client-1.1.54236-4.el7.x86_64 6/8 Installing : clickhouse-compressor-1.1.54236-4.el7.x86_64 7/8 Installing : clickhouse-debuginfo-1.1.54236-4.el7.x86_64 8/8 Verifying : clickhouse-debuginfo-1.1.54236-4.el7.x86_64 1/8 Verifying : libtool-ltdl-2.4.2-21.el7_2.x86_64 2/8 Verifying : unixODBC-2.3.1-11.el7.x86_64 3/8 Verifying : libicu-50.1.2-15.el7.x86_64 4/8 Verifying : clickhouse-server-1.1.54236-4.el7.x86_64 5/8 Verifying : clickhouse-client-1.1.54236-4.el7.x86_64 6/8 Verifying : clickhouse-compressor-1.1.54236-4.el7.x86_64 7/8 Verifying : clickhouse-server-common-1.1.54236-4.el7.x86_64 8/8 Installed: clickhouse-client.x86_64 0:1.1.54236-4.el7 clickhouse-compressor.x86_64 0:1.1.54236-4.el7 clickhouse-debuginfo.x86_64 0:1.1.54236-4.el7 clickhouse-server.x86_64 0:1.1.54236-4.el7 clickhouse-server-common.x86_64 0:1.1.54236-4.el7 libtool-ltdl.x86_64 0:2.4.2-21.el7_2 unixODBC.x86_64 0:2.3.1-11.el7 Dependency Installed: libicu.x86_64 0:50.1.2-15.el7 Complete! [root@yinzhengjie clickhouse]#
三.配置clickhouse
1>.备份clickhouse-server配置文件config.xml
[root@yinzhengjie ~]# cp /etc/clickhouse-server/config.xml /etc/clickhouse-server/config.xml.`date +%F` [root@yinzhengjie ~]#
2>.修改clickhouse-server配置文件config.xml（指定clickhouse-server的主机地址）
[root@yinzhengjie ~]# sed -i 's#<listen_host>127.0.0.1</listen_host>#<listen_host>10.1.2.105</listen_host>#' /etc/clickhouse-server/config.xml [root@yinzhengjie ~]#
3>.修改clickhouse-server配置文件config.xml（修改默认的9000端口，我们自定义该服务的端口为9999）
[root@yinzhengjie ~]# sed -i 's#<tcp_port>9000</tcp_port>#<tcp_port>9999</tcp_port>#' /etc/clickhouse-server/config.xml [root@yinzhengjie ~]#
4>.备份clickhouse-server配置文件users.xml
[root@yinzhengjie ~]# cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users.xml.`date +%F` [root@yinzhengjie ~]#
5>.修改clickhouse-server配置文件users.xml（修改/etc/clickhouse-server/users.xml配置文件的第83行，改地址改为你当前部署的服务器IP地址，不推荐使用主机名！下图是我修改后的值）
四.clickhouse管理方式
1>.启动clickhouse
[root@yinzhengjie ~]# clickhouse-server --config-file=/etc/clickhouse-server/config.xml Include not found: clickhouse_remote_servers Include not found: clickhouse_compression 2018.10.10 20:37:20.997594 [ 1 ] <Warning> Application: Logging to console 2018.10.10 20:37:21.000228 [ 1 ] <Information> : Starting daemon with revision 54236 2018.10.10 20:37:21.000337 [ 1 ] <Information> Application: starting up 2018.10.10 20:37:21.001299 [ 1 ] <Debug> Application: Set max number of file descriptors to 4096 (was 1024). 2018.10.10 20:37:21.001317 [ 1 ] <Debug> Application: Initializing DateLUT. 2018.10.10 20:37:21.001326 [ 1 ] <Trace> Application: Initialized DateLUT with time zone `Asia/Shanghai'. 2018.10.10 20:37:21.048483 [ 1 ] <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'yinzhengjie' as replica host. 2018.10.10 20:37:21.048676 [ 1 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/users.xml' 2018.10.10 20:37:21.053081 [ 1 ] <Warning> ConfigProcessor: Include not found: networks 2018.10.10 20:37:21.053114 [ 1 ] <Warning> ConfigProcessor: Include not found: networks 2018.10.10 20:37:21.054194 [ 1 ] <Information> Application: Loading metadata. 2018.10.10 20:37:21.055243 [ 1 ] <Information> DatabaseOrdinary (default): Total 0 tables. 2018.10.10 20:37:21.055919 [ 1 ] <Debug> Application: Loaded metadata. 2018.10.10 20:37:21.056089 [ 1 ] <Information> DatabaseOrdinary (system): Total 0 tables. 2018.10.10 20:37:21.057059 [ 1 ] <Information> Application: Listening http://[::1]:8123 2018.10.10 20:37:21.057138 [ 1 ] <Information> Application: Listening tcp: [::1]:9999 2018.10.10 20:37:21.057198 [ 1 ] <Information> Application: Listening interserver: [::1]:9009 2018.10.10 20:37:21.057289 [ 1 ] <Information> Application: Listening http://10.1.2.105:8123 2018.10.10 20:37:21.057420 [ 1 ] <Information> Application: Listening tcp: 10.1.2.105:9999 2018.10.10 20:37:21.057486 [ 1 ] <Information> Application: Listening interserver: 10.1.2.105:9009 2018.10.10 20:37:21.057937 [ 1 ] <Information> Application: Ready for connections. 2018.10.10 20:37:23.048780 [ 2 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/config.xml' 2018.10.10 20:37:23.053196 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_remote_servers 2018.10.10 20:37:23.053332 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_compression
2>.查看clickhouse服务是否启动
[root@yinzhengjie ~]# netstat -untalp | grep 9999 tcp 0 0 10.1.2.105:9999 0.0.0.0:* LISTEN 23259/clickhouse-se tcp6 0 0 ::1:9999 :::* LISTEN 23259/clickhouse-se [root@yinzhengjie ~]#
3>.使用客户端链接测试
[root@yinzhengjie ~]# clickhouse-client --host=10.1.2.105 --port=9999 ClickHouse client version 1.1.54236. Connecting to 10.1.2.105:9999. Connected to ClickHouse server version 1.1.54236. :) :) select now(); SELECT now() ┌───────────────now()─┐ │ 2018-10-10 20:44:14 │ └─────────────────────┘ 1 rows in set. Elapsed: 0.067 sec. :) :) show tables; SHOW TABLES Ok. 0 rows in set. Elapsed: 0.003 sec. :) :)
4>.关闭clickhouse服务
[root@yinzhengjie ~]# ps -aux|grep clickhouse-server | grep -v grep root 24998 0.1 0.3 319708 43372 pts/1 Sl+ 20:48 0:00 clickhouse-server --config-file=/etc/clickhouse-server/config.xml [root@yinzhengjie ~]# [root@yinzhengjie ~]# kill -9 `ps -aux|grep clickhouse-server | grep -v grep | awk '{print $2}'` [root@yinzhengjie ~]# You have new mail in /var/spool/mail/root [root@yinzhengjie ~]# [root@yinzhengjie ~]# ps -aux|grep clickhouse-server | grep -v grep [root@yinzhengjie ~]# [root@yinzhengjie ~]#
5>.为clickhouse-server添加守护进程
[root@yinzhengjie ~]# nohup clickhouse-server --config-file=/etc/clickhouse-server/config.xml >/var/log/yinzhengjie-clickhouse.log 2>&1 & #会在"/var/log"目录下生成一个名称为“yinzhengjie-clickhouse.log”的日志文件 [1] 26619 [root@yinzhengjie ~]# [root@yinzhengjie ~]# tail -10f /var/log/yinzhengjie-clickhouse.log 2018.10.10 20:58:53.981429 [ 1 ] <Information> Application: Listening http://[::1]:8123 2018.10.10 20:58:53.981479 [ 1 ] <Information> Application: Listening tcp: [::1]:9999 2018.10.10 20:58:53.981516 [ 1 ] <Information> Application: Listening interserver: [::1]:9009 2018.10.10 20:58:53.981554 [ 1 ] <Information> Application: Listening http://10.1.2.105:8123 2018.10.10 20:58:53.981590 [ 1 ] <Information> Application: Listening tcp: 10.1.2.105:9999 2018.10.10 20:58:53.981624 [ 1 ] <Information> Application: Listening interserver: 10.1.2.105:9009 2018.10.10 20:58:53.981796 [ 1 ] <Information> Application: Ready for connections. 2018.10.10 20:58:55.973241 [ 2 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/config.xml' 2018.10.10 20:58:55.977391 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_remote_servers 2018.10.10 20:58:55.977455 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_compression 2018.10.10 20:59:09.185410 [ 3 ] <Trace> TCPConnectionFactory: TCP Request. Address: 10.1.2.105:41874 2018.10.10 20:59:09.185660 [ 3 ] <Debug> TCPHandler: Connected ClickHouse client version 1.1.54236, user: default. 2018.10.10 20:59:18.439436 [ 3 ] <Debug> executeQuery: (from 10.1.2.105:41874) select now() 2018.10.10 20:59:18.440437 [ 3 ] <Trace> InterpreterSelectQuery: FetchColumns -> Complete 2018.10.10 20:59:18.440867 [ 3 ] <Debug> executeQuery: Query pipeline: Expression Expression One 2018.10.10 20:59:18.441657 [ 3 ] <Information> executeQuery: Read 1 rows, 1.00 B in 0.002 sec., 510 rows/sec., 510.97 B/sec. 2018.10.10 20:59:18.441764 [ 3 ] <Debug> MemoryTracker: Peak memory usage (for query): 1.00 MiB. 2018.10.10 20:59:18.441804 [ 3 ] <Debug> MemoryTracker: Peak memory usage (for user): 1.00 MiB. 2018.10.10 20:59:18.441849 [ 3 ] <Debug> MemoryTracker: Peak memory usage (total): 1.00 MiB. 2018.10.10 20:59:18.441882 [ 3 ] <Information> TCPHandler: Processed in 0.003 sec.
好啦～分享就到这里了，时间过的咋就这么快呢？一不小心就晚上九点多了，关于ClickHouse集群的部署等我后期研究好了也会分享给大家的，我得赶紧走了，不然赶不上地铁啦！晚安，兄弟们！
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
欢迎交流学习技术交流，个人微信: "JasonYin2020"（添加时请备注来源及意图备注）
作者: 尹正杰, 博客: https://www.cnblogs.com/yinzhengjie/p/9768575.html
