mysql Last_SQL_Errno: 1197 Coordinator stopped because there were error(s) in the worker(s)问题处理
Last_SQL_Errno: 1197 Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1103545303. See error log and/or performance_schema.replication_applier_status_by_worker table for more details
mysql报警主从不同步，查看发现是sql_running断了
问题分析：
有一个aiticls表大概有20W条数据开发要对它进行操作，提前在主库直接复制备份了一份
可能是数据量太大，一时无法处理
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: cmsmarketing.wai.prod.mysql.ws
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000046
Read_Master_Log_Pos: 9219397
Relay_Log_File: mysql-relay-bin.000199
Relay_Log_Pos: 444825125
Relay_Master_Log_File: mysql-bin.000045
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB: information_schema,performance_schema,undolog,for_nagios,information_schema,performance_schema,undolog,for_nagios
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: information_schema.%,performance_schema.%,information_schema.%,performance_schema.%
Last_Errno: 1197
Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1103545303. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Skip_Counter: 0
Exec_Master_Log_Pos: 969112684
Relay_Log_Space: 2692467349
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1197
Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1103545303. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Replicate_Ignore_Server_Ids:
Master_Server_Id: 16
Master_UUID: bda16924-6fe2-11e8-9376-005056912d4a
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 190830 00:31:48
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
mysql> show global variables like '%binlog_cache%';
+-----------------------+-----------+
| Variable_name | Value |
+-----------------------+-----------+
| binlog_cache_size | 65536 |
| max_binlog_cache_size | 134217728 |
+-----------------------+-----------+
2 rows in set (0.01 sec)
mysql> show global status like '%binlog_cache%';
+-----------------------+---------+
| Variable_name | Value |
+-----------------------+---------+
| Binlog_cache_disk_use | 266 |
| Binlog_cache_use | 3186562 |
+-----------------------+---------+
2 rows in set (0.00 sec)
mysql> set @@global.max_binlog_cache_size=500000000;
Query OK, 0 rows affected, 1 warning (0.00 sec)
stop slave
start slave
过了一会再次报错：
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: cmsmarketing.wai.prod.mysql.ws
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000046
Read_Master_Log_Pos: 10459073
Relay_Log_File: mysql-relay-bin.000199
Relay_Log_Pos: 444825125
Relay_Master_Log_File: mysql-bin.000045
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB: information_schema,performance_schema,undolog,for_nagios,information_schema,performance_schema,undolog,for_nagios
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: information_schema.%,performance_schema.%,information_schema.%,performance_schema.%
Last_Errno: 1197
Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1469407519. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Skip_Counter: 0
Exec_Master_Log_Pos: 969112684
Relay_Log_Space: 2693707771
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1197
Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1469407519. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Replicate_Ignore_Server_Ids:
Master_Server_Id: 16
Master_UUID: bda16924-6fe2-11e8-9376-005056912d4a
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Waiting for workers to exit
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 190830 00:38:37
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
# 把处理的cache_size配置为1G，问题解决
>set global max_binlog_cache_size=10737418240;
另外一个机房的从库报错记录：
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: cms_marketing.wai.prod.mysql.eus
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000013
Read_Master_Log_Pos: 7477932
Relay_Log_File: mysql-relay-bin.000064
Relay_Log_Pos: 36069896
Relay_Master_Log_File: mysql-bin.000012
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB: information_schema,performance_schema,undolog,for_nagios,information_schema,performance_schema,undolog,for_nagios
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: information_schema.%,performance_schema.%,information_schema.%,performance_schema.%
Last_Errno: 1197
Last_Error: Could not execute Write_rows event on table cms_marketing.articles_copy1; Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage; increase this mysqld variable and try again, Error_code: 1197; Writing one row to the row-based binary log failed, Error_code: 1534; handler error HA_ERR_RBR_LOGGING_FAILED; the event's master log mysql-bin.000012, end_log_pos 336399959
Skip_Counter: 0
Exec_Master_Log_Pos: 202043090
Relay_Log_Space: 1922987726
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1197
Last_SQL_Error: Could not execute Write_rows event on table cms_marketing.articles_copy1; Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage; increase this mysqld variable and try again, Error_code: 1197; Writing one row to the row-based binary log failed, Error_code: 1534; handler error HA_ERR_RBR_LOGGING_FAILED; the event's master log mysql-bin.000012, end_log_pos 336399959
Replicate_Ignore_Server_Ids:
Master_Server_Id: 164
Master_UUID: 968e1d7f-957c-11e8-b650-00163e00923d
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 190830 00:47:25
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
