오픈스택 PortBindingFailed 에러 처리

| 2021년 6월 3일 | 0 Comments

최근 오픈 스택 인스턴스 빌드 테스트시 컴퓨트 노드 nova, neutron 로그에 PortBindingFailed 에러와 함께 서버 생성이 실패하는 현상이 발생했습니다.

서버는 정상적으로 인스턴스 생성되던 서버였으며 vm도 이미 여러대 생성되있는 상태였습니다.

테스트 OS는 Ubuntu 18.04 LTS에 오픈스택 버전 Queens이며 dvr 구성되어 있습니다.

해결 방법은 ovsdb-server 재시작후 가상 브릿지 재생성후 관련 서비스 재시작후 정상화 되었습니다.

 


 

  • 서버 생성

    # openstack server create –image Ubuntu_20.04_x86 –flavor test –net yhtest –availability-zone test:compute jyh

    +————————————-+———————————————————+

    | Field | Value |

    +————————————-+———————————————————+

    | OS-DCF:diskConfig | MANUAL |

    | OS-EXT-AZ:availability_zone | test |

    | OS-EXT-SRV-ATTR:host | None |

    | OS-EXT-SRV-ATTR:hypervisor_hostname | None |

    | OS-EXT-SRV-ATTR:instance_name | |

    | OS-EXT-STS:power_state | NOSTATE |

    | OS-EXT-STS:task_state | scheduling |

    | OS-EXT-STS:vm_state | building |

    | OS-SRV-USG:launched_at | None |

    | OS-SRV-USG:terminated_at | None |

    ..
    ..

  • 에러 내용

    nova.log

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [req-92826ae4-3562-4983-83d7-375eb8280446 3f2299fa330a4689acd1c77cfba32cc1 6b694b8152e347d58037098ae18d4ad5 – default default] [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] Instance failed to spawn: PortBindingFailed: Binding failed for port 81c2538e-8436-4be5-bc42-109d5c84939f, please check neutron logs for more information.

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] Traceback (most recent call last):

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/compute/manager.py”, line 2273, in _build_resources

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] yield resources

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/compute/manager.py”, line 2053, in _build_and_run_instance

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] block_device_info=block_device_info)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py”, line 3117, in spawn

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] mdevs=mdevs)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py”, line 5401, in _get_guest_xml

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] network_info_str = str(network_info)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/model.py”, line 568, in __str__

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] return self._sync_wrapper(fn, *args, **kwargs)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/model.py”, line 551, in _sync_wrapper

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] self.wait()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/model.py”, line 583, in wait

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] self[:] = self._gt.wait()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/eventlet/greenthread.py”, line 175, in wait

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] return self._exit_event.wait()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/eventlet/event.py”, line 121, in wait

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] return hubs.get_hub().switch()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py”, line 294, in switch

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] return self.greenlet.switch()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/eventlet/greenthread.py”, line 214, in main

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] result = function(*args, **kwargs)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/utils.py”, line 906, in context_wrapper

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] return func(*args, **kwargs)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/compute/manager.py”, line 1452, in _allocate_network_async

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] six.reraise(*exc_info)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/compute/manager.py”, line 1435, in _allocate_network_async

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] bind_host_id=bind_host_id)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py”, line 989, in allocate_for_instance

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] bind_host_id, available_macs, requested_ports_dict)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py”, line 1122, in _update_ports_for_instance

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] vif.destroy()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py”, line 220, in __exit__

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] self.force_reraise()

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py”, line 196, in force_reraise

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] six.reraise(self.type_, self.value, self.tb)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py”, line 1092, in _update_ports_for_instance

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] port_client, instance, port_id, port_req_body)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py”, line 470, in _update_port

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] _ensure_no_port_binding_failure(port)

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] File “/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py”, line 215, in _ensure_no_port_binding_failure

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] raise exception.PortBindingFailed(port_id=port[‘id’])

    2021-06-03 09:21:59.416 12204 ERROR nova.compute.manager [instance: c8fe3174-dcb3-418c-9b2a-ba5b0d6bf969] PortBindingFailed: Binding failed for port 81c2538e-8436-4be5-bc42-109d5c84939f, please check neutron logs for more information.

    neutron-openvswitch-agent 로그

    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl [-] Post-commit checks failed: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.AddBridgeCommand object at 0x7f497d017450>, <ovsdbapp.backend.ovs_idl.command.DbAddCommand object at 0x7f497d017510>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7f497d017550>] exceeded timeout 10 seconds
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl Traceback (most recent call last):
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl File “/usr/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py”, line 39, in post_commit
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl self.do_post_commit(txn)
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl File “/usr/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py”, line 59, in do_post_commit
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl timeout=self.timeout)
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.AddBridgeCommand object at 0x7f497d017450>, <ovsdbapp.backend.ovs_idl.command.DbAddCommand object at 0x7f497d017510>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7f497d017550>] exceeded timeout 10 seconds
    2021-06-03 09:21:57.624 12333 ERROR ovsdbapp.schema.open_vswitch.impl_idl
    2021-06-03 09:21:57.626 12333 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Commands [<ovsdbapp.schema.open_vswitch.commands.AddBridgeCommand object at 0x7f497d017450>, <ovsdbapp.backend.ovs_idl.command.DbAddCommand object at 0x7f497d017510>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7f497d017550>] exceeded timeout 10 seconds Agent terminated!: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.AddBridgeCommand object at 0x7f497d017450>, <ovsdbapp.backend.ovs_idl.command.DbAddCommand object at 0x7f497d017510>, <ovsdbapp.backend.ovs_idl.command.DbSetCommand object at 0x7f497d017550>] exceeded timeout 10 seconds

     

  • 처리 내역

    # service ovsdb-server status

    ovsdb-server.service – Open vSwitch Database Unit

    Loaded: loaded (/lib/systemd/system/ovsdb-server.service; static; vendor preset: enabled)

    Active: active (running) since Thu 2019-10-24 10:56:57 KST; 1 years 7 months ago

    Tasks: 1 (limit: 4915)

    CGroup: /system.slice/ovsdb-server.service

    └─2583 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info –remote=punix:/var/run/openvswitch/db.sock –private-key=db:Open_vSwitch,SSL,private_key –certificate=db:Open_vSw

    Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

    # service ovsdb-server restart

    # ovs-vsctl del-br {스위치}
    # ovs-vsctl add-br {스위치}
    # ovs-vsctl add-port {스위치} {연결할 인터페이스}

    # systemctl restart neutron-openvswitch-agent neutron-l3-agent networking

    ※ 운영을 mgmt, internal, public 하나의 인터페이스로 운영시 네트웍 끊길수 있기 때문에 콘솔로 안전하게 작업하시기 바랍니다.

Category: 가상화/클라우드

Jang Smile

About the Author ()