오픈스택 퀸즈 메세지 서비스 에러 픽스

| 2021년 2월 3일 | 0 Comments

최근 일주일 사이 오픈스택 퀸즈 버전 메세지 서비스에서 에러가 대량 발생하고 새로운 노드 추가가 안되는 문제점이 발생 하였습니다.

확인중에 python-oslo.messaging 패키지 버전이 업데이트 된것을 확인 하였으며 이로 인해 문제가 발생된것이라 추측이 가능 했습니다..

문제 해결은 패키지 버전을 낮춰서 설치하고 문제점들이 해결되었습니다.

 


 

  • nova-compute 로그
    # tail -f /var/log/nova/nova-compute.log
    2021-02-03 14:44:31.295 2923 WARNING nova.conductor.api [req-a7ebacdc-4bf4-4925-9336-ab0cebc050d4 – – – – -] Timed out waiting for nova-conductor. Is it running?
    Or did this service start before nova-conductor?
    Reattempting establishment of nova-conductor connection…: MessagingTimeout: Timed out waiting for a reply to message ID 2d465c45f8aa46ca926ced10bb92f99b
    2021-02-03 14:45:31.300 2923 WARNING nova.conductor.api [req-a7ebacdc-4bf4-4925-9336-ab0cebc050d4 – – – – -] Timed out waiting for nova-conductor. Is it running?
    Or did this service start before nova-conductor?
    Reattempting establishment of nova-conductor connection…: MessagingTimeout: Timed out waiting for a reply to message ID 0dddee32759043ad95d1d70aa5a66fd7
  • neutron 로그
    2021-02-03 14:39:46.811 10583 ERROR neutron.common.rpc [req-8b96232d-5147-4537-86fb-a5a62cf081b2 – – – – -] Timeout in RPC method get_dvr_mac_address_by_host.
    Waiting for 52 seconds before next attempt.
    If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
    : MessagingTimeout: Timed out waiting for a reply to message ID 36b9498924354183bc7d017e125c0ab9
  • nova-conductor 로그
    2021-02-03 14:50:52.789 1720 ERROR oslo.messaging._drivers.impl_rabbit [req-add4e2aa-6941-4cb7-b633-b52b51a9184d – – – – -] [a7ac2e0a-014e-4870-9224-d1687cd243e9]
    AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out
  • python-oslo.messaging 버전 확인
    # dpkg -l |grep oslo.messaging
    ii python-oslo.messaging 5.35.0-0ubuntu2 all oslo messaging library – Python 2.x
  • 패키지 삭제

    # apt-cache policy python-oslo.messaging
    python-oslo.messaging:
    Installed: 5.35.0-0ubuntu2
    Candidate: 5.35.0-0ubuntu2
    Version table:
    *** 5.35.0-0ubuntu2 500
    500 http://kr.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
    500 http://kr.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages
    100 /var/lib/dpkg/status
    5.35.0-0ubuntu1 500
    500 http://kr.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
    500 http://kr.archive.ubuntu.com/ubuntu bionic/main i386 Packages

    # apt remove python-oslo.messaging

  • 낮은 버전 패키지 설치
    # apt install python-oslo.messaging=5.35.0-0ubuntu1
  • 오픈스택 서비스 패키지 재설치
    # apt install nova-compute neutron-openvswitch-agent
  • 오픈 스택 서비스 확인

    # openstack compute service list -f value –service nova-compute –host compute1004
    21 nova-compute compute1004 nova enabled up 2021-02-03T06:45:20.000000

    # tail -f /var/log/nova/nova-compute.log

    2021-02-03 15:07:23.567 18317 INFO nova.scheduler.client.report [req-02526fbf-d566-4f88-ae97-e9ace2ddbcf0 – – – – -]
    [req-c1727932-29d6-452b-a9e0-a84a6b3564ec] Created resource provider record via placement API for resource provider with UUID b932dd2c-5a34-41f2-b2d0-14bc0d5e481f and name compute1004.

    2021-02-03 15:08:21.280 18317 INFO nova.compute.manager [req-1f857aef-4b32-4baf-a52c-266df473d49c – – – – -] Running instance usage audit for host compute1004 from 2021-02-03 05:00:00
    to 2021-02-03 06:00:00. 0 instances.

    2021-02-03 15:08:21.495 18317 INFO nova.compute.resource_tracker [req-1f857aef-4b32-4baf-a52c-266df473d49c – – – – -]
    Final resource view: name=compute1004 phys_ram=32128MB used_ram=512MB phys_disk=116GB used_disk=0GB total_vcpus=8 used_vcpus=0 pci_stats=[]

Category: 가상화/클라우드

Jang Smile

About the Author ()