Apache Flink 1.16.0部署手册
1 参考文献
Apache Flink官网：https://flink.apache.org/downloads.html
2 硬件环境部署
2.1 系统配置说明
| 系统 | CPU | 内存 | 硬盘 | IP |
|---|---|---|---|---|
| Ubuntu 22.04.1 LTS | 4 | 16G | 200G | 192.168.100.101 |
3 软件环境部署
3.1 软件环境清单
#JDK：1.8下载地址：
http://www.oracle.com/technetwork/java/javase/downloads/index.html
#MySQL：8.0.30下载地址：
https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.30-linux-glibc2.12-x86_64.tar.xz
##Flink：1.16.0下载地址:
https://archive.apache.org/dist/flink/flink-1.16.0/flink-1.16.0-bin-scala_2.12.tgz
3.2 软件环境部署顺序
JDK--》MySQL--》Apache Flink
3.3 JDK部署
cd /data
tar -zxvf jdk-8u341-linux-x64.tar.gz -C /usr/local/
vim /etc/profile
#jdk
export JAVA_HOME=/usr/local/jdk1.8.0_341/
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
#添加后保存退出
source /etc/profile
3.4 数据库安装
安装MySQL
cd /data
tar -xvf mysql-8.0.30-linux-glibc2.12-x86_64.tar.xz
mv mysql-8.0.30-linux-glibc2.12-x86_64 /usr/local/mysql
groupadd mysql
useradd -r -g mysql -s /sbin/nologin mysql
mkdir /data/mysql -p
mkdir /usr/local/mysql/etc -p
mkdir /var/run/mysql/
chown -R mysql:mysql /usr/local/mysql/
chown -R mysql:mysql /data/mysql
chown -R mysql:mysql /var/run/mysql/
编辑配置
vim /usr/local/mysql/etc/my.cnf
[client]
port=3306
socket=/var/run/mysql/mysql.sock
[mysqld]
user=mysql
port=3306
bind-address=0.0.0.0
datadir=/data/mysql
basedir=/usr/local/mysql
socket=/var/run/mysql/mysql.sock
pid-file=/var/run/mysql/mysql.pid
default-storage-engine=INNODB
character_set_server=utf8mb4
lower_case_table_names=1
table_open_cache=128
max_connections=1200
max_connect_errors=1000
innodb_file_per_table=1
innodb_buffer_pool_size=1G
max_allowed_packet=64M
transaction_isolation=READ-COMMITTED
innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout=1800
innodb_flush_log_at_trx_commit=0
sync_binlog=0
symbolic-links=0
explicit_defaults_for_timestamp=true
server-id=1
log-error=/data/mysql/mysql.err
log-bin=mysql-bin
expire_logs_days = 7
binlog_format=mixed
slow_query_log=1
long_query_time=10
slow_query_log_file=/data/mysql/mysql-slow.log
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
skip-name-resolve
[mysql]
default-character-set=utf8mb4
[mysql.server]
default-character-set=utf8mb4
初始化数据
cd /usr/local/mysql/bin
./mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/data/mysql --lower-case-table-names=1
tailf /data/mysql/mysql.err
cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
/etc/init.d/mysqld start
echo 'PATH=$PATH:$HOME/bin:/usr/local/mysql/bin' >> /etc/profile
source /etc/profile
登录MySQL修改权限
mysql -uroot -p
mysql> alter user 'root'@'localhost' identified by 'sZvoOlW3R4rqckQjXtKn';
mysql> use mysql;
mysql> select host, user, authentication_string, plugin from user;
mysql> create user 'root'@'%' identified by 'sZvoOlW3R4rqckQjXtKn';
mysql> grant all privileges on *.* to 'root'@'%' with grant option;
mysql> select host, user, authentication_string, plugin from user;
3.5 其他支撑组件安装
安装好的lib中只有flink的jar包，如果提交的job有用到连接mysql或者使用flink cdc，需要把相应的驱动加入到lib中:
mysql-connector-java-8.0.16.jar
flink-connector-jdbc_2.12-1.14.5.jar
flink-sql-connector-mysql-cdc-2.2.1.jar
##连接mysql提前下载mysql-connector-java-8.0.16.jar驱动即可，下载地址：
https://repo1.maven.org/maven2/mysql/mysql-connector-java/
4 系统安装与配置
4.1 解压二进制包
cd /data/
tar -zxvf flink-1.16.0-bin-scala_2.12.tgz
mv flink-1.16.0 flink
4.2 修改主配置文件
备份yaml文件
cd /data/flink
cp conf/flink-conf.yaml conf/flink-conf.yaml.bak
修改flink-conf.yaml
vim conf/flink-conf.yaml
##修改以下配置：
jobmanager.rpc.address: 0.0.0.0
jobmanager.bind-host: 0.0.0.0
taskmanager.bind-host: 0.0.0.0
taskmanager.host: 0.0.0.0
rest.address: 192.168.100.101
rest.bind-address: 192.168.100.101
4.3 添加MySQL驱动
mysql-connector-java-8.0.16.jar驱动添加到/data/flink/lib/目录下即可。
5 系统初始化与确认
5.1 数据库初始化
登录MySQL添加flink用户并赋权
mysql> CREATE DATABASE flink DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE USER 'flink'@'%' IDENTIFIED BY 'juFb493esiQV!rMBZhoyT';
mysql> GRANT ALL PRIVILEGES ON flink.* TO 'flink'@'%';
mysql> FLUSH PRIVILEGES;
mysql> exit
5.2 系统部署确认
启动Flink服务
cd /data/flink/
./bin/start-cluster.sh
5.3 加入开机自启
由于Ubuntu不再inited管理系统,改用 systemd，没有/etc/rc.local文件，所以需要自己配置rc.local；在/usr/lib/systemd/system/rc-local.service文件中添加下面内容：
vim /usr/lib/systemd/system/rc-local.service
[Install]
WantedBy=multi-user.target
Alias=rc-local.service
重新加载文件
systemctl daemon-reload
新建/etc/rc.local文件，如果用户默认运行环境在 /bin/bash 下，则应修改#!bin/sh -e为 #!bin/bash -e 。自行添加的命令要在exit 0前面。
vim /etc/rc.local
#!/bin/bash -e
# rc.local
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
# In order to enable or disable this script just change the execution
# bits.
# By default this script does nothing.
export JAVA_HOME=/usr/local/jdk1.8.0_341
su - root -c "bash /data/flink/bin/start-cluster.sh"
exit 0
修改权限
chmod 755 /etc/rc.local
重启rc服务，并加入开机自启
systemctl restart rc-local.service
systemctl enable rc-local
5.4 访问确认
浏览器访问地址 http://localhost:8081 即可登录系统UI。（无账户密码）
6 问题处理
暂无
