openstack cinder状态错误

cinder service状态错误

  日常巡检发现cinder service state出现down，提示丢失数据库连接错误，导致了cinder服务状态更新或者创建卷报错。

问题发现

$ openstack volume service list
+------------------+-------------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                                | Zone | Status  | State | Updated At                 |
+------------------+-------------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | cloudHHvh2.vim1.local | nova | enabled | down   | 2022-08-12T03:20:53.000000 |
| cinder-scheduler | cloudsHHvh1.vim1.local | nova | enabled | down    | 2022-08-12T03:20:56.000000 |
| cinder-volume    | control@fusionstorage  | nova | enabled | down    | 2022-08-12T03:20:51.000000 |
| cinder-scheduler | cloudsHHvh3.vim1.local | nova | enabled | down    | 2022-08-12T03:20:47.000000 |
+------------------+-------------------------------------+------+---------+-------+----------------------------+

查看cinder日志信息，发现都是连接数据库问题的错误

2022-08-12 10:14:12.246 6 ERROR cinder   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py",line 687, in _read_bytes()
DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'UPDATE services SET updated_at=%(updated_at)s, rpc_current_version=%(rpc_current_version)s, object_current_version=%(object_current_version)s WHERE services.deleted = false AND services.id = %(id_1)s'] [parameters: {'object_current_version': u'1.21', 'rpc_current_version': u'3.5', u'id_1': 4, 'updated_at': datetime.datetime(2022, 8, 12, 1, 14, 12, 243742)}]
2022-08-12 10:14:12.246 6 ERROR cinder Traceback (most recent call last):
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/bin/cinder-scheduler", line 10, in <module>
2022-08-12 10:14:12.246 6 ERROR cinder     sys.exit(main())
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/cmd/scheduler.py", line 53, in main
2022-08-12 10:14:12.246 6 ERROR cinder     server = service.Service.create(binary='cinder-scheduler')
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 425, in create
2022-08-12 10:14:12.246 6 ERROR cinder     cluster=cluster)
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 193, in __init__
2022-08-12 10:14:12.246 6 ERROR cinder     service_ref.save()
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 151, in save
2022-08-12 10:14:12.246 6 ERROR cinder     db.service_update(self._context, self.id, updates)
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 142, in service_update
2022-08-12 10:14:12.246 6 ERROR cinder     return IMPL.service_update(context, service_id, values)
2022-08-12 10:14:12.246 6 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 178, in wrapper
2022-08-12 10:14:12.246 6 ERROR cinder     return f(*args, **kwargs)

分析报错的代码

$ docker exec -it -u0 cinder_volume bash
$ vi /var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py +687
...
    687     def _read_bytes(self, num_bytes):
    688         self._sock.settimeout(self._read_timeout)
    689         while True:
    690             try:
    691                 data = self._rfile.read(num_bytes)
    692                 break
    693             except (IOError, OSError) as e:
    694                 if e.errno == errno.EINTR:
    695                     continue
    696                 self._force_close()
    697                 raise err.OperationalError(
    698                     CR.CR_SERVER_LOST,
    699                     "Lost connection to MySQL server during query (%s)" % (e,))
    700             except BaseException:
    701                 # Don't convert unknown exception to MySQLError.
    702                 self._force_close()
    703                 raise
                  # 如果MySQL服务器没有根据协议所述发送足够的数据，就会抛出如下错误。
    704         if len(data) < num_bytes:
    705             self._force_close()
    706             raise err.OperationalError(
    707                 CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
    708         return data
...

解决办法

  MySQL数据库中，max_allowed_packet允许的server段接受的数据包最大为4M，有的时候大的插入和更新可能会被max_allowed_packet参数限制，导致数据库更新失败，这里我们修改了max_allowed_packet参数。

$ vim /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2
[server]
#增加如下行
max_allowed_packet= '32M'

$ kolla-ansible -i /etc/ansible/hosts -t mariadb deploy