利用Go实现MySQL binlog与 es数据同步

1.安装go

go官网：https://golang.google.cn/dl/
注意！ 安装go的同时也需要安装git

wget https://golang.google.cn/dl/go1.15.2.linux-amd64.tar.gz
tar -C /usr/local -zxvf go1.15.2.linux-amd64.tar.gz

配置环境变量vim /etc/profile

export GOROOT=/usr/local/go
export GOPATH=/data/bogiang/go
export PATH=$PATH:/usr/local/go/bin

2.修改MySQL binlog日志模式

需要值得注意的是， go-mysql-elasticsearch 支持binlog的日志模式为row模式因此建议在配置MySQLbinlog的时候需指定该模式

log-bin=mysql-bin
binlog_format=row

3.go-mysql-elasticsearch安装

go env -w GOPROXY=https://goproxy.cn
go get go get github.com/siddontang/go-mysql-elasticsearch
cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch
make

配置river.toml

#  MySQL 配置：地址，用户名，密码
my_addr = "xxxx:3306"
my_user = "xxxx"
my_pass = "xxxx"

# Elasticsearch地址
es_addr = "xxxxx"

# 存储数据的位置
data_dir = "/var"

# Inner Http status address
stat_addr = "xxxxx:12800"
stat_path = "/es"
# pseudo server id like a slave
server_id = 1
# mysql or mariadb
flavor = "mysql"
# mysqldump execution path
# mysqldump = "mysqldump"
# minimal items to be inserted in one bulk
bulk_size = 128

# force flush the pending requests if we don't have enough items >= bulk_size
flush_bulk_time = "200ms"

# Ignore table without primary key
skip_no_pk_table = false

# elasticsearch 与 mysql 同步时对应的数据库名称
# mysql的数据源
[[source]]
schema = "lw"
tables = ["goods"]

# es 映射的mapping
[[rule]]
schema = "lw"
table = "goods"
# es的索引名
index = "lw_goods"
type = "link_info"

启动服务

$GOPATH/src/github.com/siddontang/go-mysql-elasticsearch/bin/go-mysql-elasticsearch -config=/xxx/river.toml