openstack queens ‘DBConnectionError’ 로그 출력 되지 않게 설정하기

| 2019년 10월 14일 | 0 Comments

오픈 스택 queens 버전을 설치하고 운영을 하다보면 오픈 스택 서비스 모든 로그에 DBConnectionError 에러가 발생하는것을 볼수 있다.

분명 정상적으로 호출이 이루어지고 에러가 발생하지 않는데 log 파일들을 보면 ‘Lost connection to MySQL server during query’ 이런 메시지가 쌓인다.

예전 버전 mitaka에서는 발생하지 않는 부분이다.

해당 문제는 오픈스택 소스에서 mysql 연결 관련 timeout 값과 관련되어 보이며 해결은 mysql 설정 파일에 wait_timeout, connect_timeout 값을 수정후 해당 에러가 발생하지 않는다.

 

  • nova-api 에러 메시지
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines [req-cdccc42a-d4be-49ce-a735-ca57891cf39b 1a90b977f0e24e4c9b6a6d2b9926d4d3 c45b3ac84505413cafd1e3a7fe283a99 – default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, ‘Lost connection to MySQL server during query’) [SQL: u’SELECT 1′]
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py”, line 73, in _connect_ping_listener
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1]))
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py”, line 877, in scalar
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar()
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py”, line 945, in execute
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py”, line 263, in _execute_on_connection
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py”, line 1053, in _execute_clauseelement
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py”, line 1189, in _execute_context
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines context)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py”, line 1398, in _handle_dbapi_exception
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py”, line 203, in raise_from_cause
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py”, line 1182, in _execute_context
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines context)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py”, line 470, in do_execute
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/cursors.py”, line 165, in execute
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/cursors.py”, line 321, in _query
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines conn.query(q)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/connections.py”, line 860, in query
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/connections.py”, line 1061, in _read_query_result
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines result.read()
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/connections.py”, line 1349, in read
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/connections.py”, line 991, in _read_packet
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines File “/usr/lib/python2.7/dist-packages/pymysql/connections.py”, line 1037, in _read_bytes
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, “Lost connection to MySQL server during query”)
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, ‘Lost connection to MySQL server during query’) [SQL: u’SELECT 1′]
    2019-10-14 09:18:07.284 9098 ERROR oslo_db.sqlalchemy.engines
  • gnocchi-metricd 에러 로그
    Traceback (most recent call last):
    File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1182, in _execute_context
    context)
    File “/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py”, line 470, in do_execute
    cursor.execute(statement, parameters)
    File “/usr/lib/python3/dist-packages/pymysql/cursors.py”, line 165, in execute
    result = self._query(query)
    File “/usr/lib/python3/dist-packages/pymysql/cursors.py”, line 321, in _query
    conn.query(q)
    File “/usr/lib/python3/dist-packages/pymysql/connections.py”, line 860, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
    File “/usr/lib/python3/dist-packages/pymysql/connections.py”, line 1061, in _read_query_result
    result.read()
    File “/usr/lib/python3/dist-packages/pymysql/connections.py”, line 1349, in read
    first_packet = self.connection._read_packet()
    File “/usr/lib/python3/dist-packages/pymysql/connections.py”, line 991, in _read_packet
    packet_header = self._read_bytes(4)
    File “/usr/lib/python3/dist-packages/pymysql/connections.py”, line 1037, in _read_bytes
    CR.CR_SERVER_LOST, “Lost connection to MySQL server during query”)
    pymysql.err.OperationalError: (2013, ‘Lost connection to MySQL server during query’)

  • mysql 설정 변경

    ..
    ..
    wait_timeout=3600
    connect_timeout = 3600
    ..
    ..

    MariaDB [(none)]> show global variables like ‘%timeout%’;
    +————————————-+———-+
    | Variable_name | Value |
    +————————————-+———-+
    | connect_timeout | 3600 |
    | deadlock_timeout_long | 50000000 |
    | deadlock_timeout_short | 10000 |
    | delayed_insert_timeout | 300 |
    | innodb_flush_log_at_timeout | 1 |
    | innodb_lock_wait_timeout | 50 |
    | innodb_print_lock_wait_timeout_info | OFF |
    | innodb_rollback_on_timeout | OFF |
    | interactive_timeout | 28800 |
    | lock_wait_timeout | 31536000 |
    | net_read_timeout | 30 |
    | net_write_timeout | 60 |
    | slave_net_timeout | 3600 |
    | thread_pool_idle_timeout | 60 |
    | wait_timeout | 3600 |
    +————————————-+———-+

     
    ※ 설정을 변경한 이후에는 mysql 포함한 openstack 모든 서비스를 재시작 이후 시점부터 로그가 쌓이지 않는다.

Category: 가상화/클라우드

장영호

About the Author ()