오픈스택 Queens VirtualInterfaceCreateException 에러 처리

| 2018년 10월 31일 | 0 Comments

오픈 스택 Queens가 릴리즈 된후 테스트중에 VirtualInterfaceCreateException: Virtual Interface creation failed 에러가 발생했습니다.

보통 설정 파일에서 잘못 기입할경우 발생하는 경우가 많은데 테스트중에 발생했으며 인스턴스가 정상적으로 설치되는 경우도 있고 dhcp 안되는 경우도 있었으며 user-data가 기능을 하지 못하는 경우도 있었습니다.

해결 하는 과정은 neutron port 에서 device_id가 reserved_dhcp_port인 포트를 삭제하고 dhcp 네임 스페이스 삭제후 서비스 재시작하고 정상화 되었습니다.

 

  • 컴퓨트 노드 에러 내용

2018-10-31 09:20:48.034 8182 INFO os_vif [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:58:19:c0,bridge_name='qbr931d9768-ac',has_traffic_filtering=True,id=931d9768-acd7-4ec1-b395-c9387592f4cc,network=Network(00279915-5d97-4c47-9416-6fe08fb5536a),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap931d9768-ac')
2018-10-31 09:20:48.200 8182 INFO nova.compute.manager [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: cd6c5b37-bb6e-4cd1-8707-eb8fd099e26c] Took 0.77 seconds to destroy the instance on the hypervisor.
2018-10-31 09:20:48.259 8182 INFO nova.virt.libvirt.driver [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: db43921b-56fa-4abe-a8ec-9d590d10b74e] Deletion of /var/lib/nova/instances/db43921b-56fa-4abe-a8ec-9d590d10b74e_del complete
2018-10-31 09:20:48.302 8182 INFO nova.virt.libvirt.driver [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: dbe3ae87-7098-499f-9d0a-a1fe53cee7a8] Deletion of /var/lib/nova/instances/dbe3ae87-7098-499f-9d0a-a1fe53cee7a8_del complete
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] Failed to allocate network(s): VirtualInterfaceCreateException: Virtual Interface creation failed
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] Traceback (most recent call last):
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2031, in _build_and_run_instance
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]     block_device_info=block_device_info)
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3089, in spawn
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]     destroy_disks_on_failure=True)
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5607, in _create_domain_and_network
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]     raise exception.VirtualInterfaceCreateException()
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] VirtualInterfaceCreateException: Virtual Interface creation failed
2018-10-31 09:20:48.349 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] 
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] Build of instance 310fd2ea-5ca5-474a-baa7-0705226219a9 aborted: Failed to allocate the network(s), not rescheduling.: BuildAbortException: Build of instance 310fd2ea-5ca5-474a-baa7-0705226219a9 aborted: Failed to allocate the network(s), not rescheduling.
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] Traceback (most recent call last):
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in _do_build_and_run_instance
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]     filter_properties, request_spec)
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2091, in _build_and_run_instance
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9]     reason=msg)
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] BuildAbortException: Build of instance 310fd2ea-5ca5-474a-baa7-0705226219a9 aborted: Failed to allocate the network(s), not rescheduling.
2018-10-31 09:20:48.368 8182 ERROR nova.compute.manager [instance: 310fd2ea-5ca5-474a-baa7-0705226219a9] 
2018-10-31 09:20:48.384 8182 INFO nova.compute.manager [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: db43921b-56fa-4abe-a8ec-9d590d10b74e] Took 0.59 seconds to destroy the instance on the hypervisor.
2018-10-31 09:20:48.443 8182 INFO nova.compute.manager [req-6dd88a35-1db0-4e5d-8604-0be89a140329 e81bdce7fb5c4f96b80e9b559ec8e190 9f3699b940a342b0860a734ee3bcea2d - default default] [instance: dbe3ae87-7098-499f-9d0a-a1fe53cee7a8] Took 0.55 seconds to destroy the instance on the hypervisor.

vif로 tap 인터페이스 생성까지 확인이 되나 vif plugging timeout 시간 600초가 지나면 에러로 빠진다.
  • 컨트롤러 노드 neutron port 상태에서 reserved_dhcp_port로 출력되는 포트 확인

root@controller:~# neutron port-list -c id -c tenant_id  --device_owner network:dhcp -c device_owner -c device_id -c fixed_ips
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+----------------------------------+--------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+
| id                                   | tenant_id                        | device_owner | device_id                                                                     | fixed_ips                                                                           |
+--------------------------------------+----------------------------------+--------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+
| 0903fbd9-3078-4c07-8dbf-b0f7c172eaa8 | 7afad4b113c543a3b249121a04f74123 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-d369b150-b696-402f-84f7-c9b9eb025d70 | {"subnet_id": "e4d9aa31-821f-4b06-a775-57dd834f2cc8", "ip_address": "10.4.0.2"}     |
| 19bd67af-090e-4253-9e4d-52d0bf38489c | b235675844294535a085282854ca20f7 | network:dhcp | reserved_dhcp_port                                                            | {"subnet_id": "402d73f2-1d69-4725-93e6-05b227f69174", "ip_address": "10.4.0.2"}     |
| 1f988237-f1ba-479c-be70-712df1b8ce88 | 8d26e59ea2654c14a71cfc242a8cb7b6 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-d6772df9-4782-4935-b844-4d1a4bc0e70c | {"subnet_id": "ea8a5adf-5ed2-42da-b13a-4db6ef834841", "ip_address": "10.4.0.2"}     |
| 23b95886-b687-4fb2-ae58-cd61c456352f | 1aff0e58bb68426b95fc4efcc0b44ca3 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-52e2afbd-10ce-465a-afbe-87636ea0b9d9 | {"subnet_id": "f8d71d05-58ee-4767-9cf0-c35ad65d7af9", "ip_address": "10.4.0.2"}     |
| 26e18a71-855d-44de-8d7a-3fe0e4b4b121 | c45b3ac84505413cafd1e3a7fe283a99 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-13c6071a-12ea-472b-993b-7f08355714f6 | {"subnet_id": "d05a5737-fc93-4bda-b3ee-4445a7f7eda8", "ip_address": "49.247.210.0"} |
| 281604a9-4fef-462d-82ca-2f2311b6decf | 9b47fa71c5bd40d5b9f9a4862796aed5 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-fd4dd3c7-2de8-4587-95c9-8f57529bbe39 | {"subnet_id": "d8836150-dc06-497a-b5d0-20cacab86bb7", "ip_address": "10.4.0.2"}     |
| 2a671b14-bea0-4b9f-a61a-0da175f0efcc | abaeaa3cdd9649f89970d423fee51e8f | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-e73c5eaf-f5f2-4d80-8711-2eb1eae32e1a | {"subnet_id": "384ae2b1-b55a-49c5-8fda-6a6fe38c46b3", "ip_address": "10.4.0.2"}     |
| 2b69c7c1-563b-442a-a2ef-4dbca3a2d07c | 15eddc717b7f4199bcbe7bc4f572681d | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-53e4b84f-7b60-467a-98a7-5388577368bc | {"subnet_id": "eaf33245-3b62-4314-ad09-494ce81ef0e6", "ip_address": "10.4.0.2"}     |
| 2e3deb54-814b-4612-b9db-d855db1ad1a6 | e4def2675648421fb60dcfd5743fcaff | network:dhcp | dhcp945e4881-9a43-5e7b-ac70-097af06eb70f-2f874cb0-fbf9-4e16-9ad4-bf7a2253b2a1 | {"subnet_id": "c9c09cd1-4645-4f0a-8791-9c6eda65a4de", "ip_address": "10.4.0.2"}     |
| 39e70972-f548-449f-871f-5cf5145d6b8e | 4c7a27a98d464bd5a1d068500364ff7e | network:dhcp | dhcp945e4881-9a43-5e7b-ac70-097af06eb70f-00279915-5d97-4c47-9416-6fe08fb5536a | {"subnet_id": "13191799-3f57-4730-a3bc-bbe94928df0e", "ip_address": "10.4.0.2"}     |
| 4a203ad0-c33d-40dd-aa76-de23d83726b1 | 718cb24c596940328c189453ecdcb12e | network:dhcp | dhcp945e4881-9a43-5e7b-ac70-097af06eb70f-d304dd65-9650-45df-8006-965efcfcf4fd | {"subnet_id": "4ed3815c-e550-4913-8f40-542123f2179a", "ip_address": "10.4.0.2"}     |
| 576a3eb6-dfb0-4296-b67f-b11ec5465cd2 | 1b22651998634efb8bc4a4d6623e7384 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-0cc5e8d6-7d7c-41a0-a88e-055c77917022 | {"subnet_id": "95b0d022-6d69-4033-bab4-967896f03fbc", "ip_address": "10.4.0.2"}     |
| 58649711-e101-46cc-a2cc-8ccdc957ee08 | 82111b411f6c4529bf280acd9541e713 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-fccc08e7-cac0-41b1-a393-895760f4783d | {"subnet_id": "035b0cce-2f02-4dd9-8a12-47a2a3981b0b", "ip_address": "10.4.0.2"}     |
| 60658c8a-df3e-4055-a569-ef5a961f1bb0 | ac686a01436d4deb9c84c14a61fdec73 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-cb0429d2-0739-48b2-8a19-4e0edb88105e | {"subnet_id": "a660feca-a5e9-4cfb-87d7-9c969ca8eed3", "ip_address": "10.4.0.2"}     |
| 64acfb66-071d-4250-bfc1-7fc804c33d99 | 8001759327cb47dfa93260154dd72502 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-52953bc3-913e-4d6b-85f5-2c3dd4ada69b | {"subnet_id": "c7f98987-1b40-4b47-9d9d-bcf423f4c678", "ip_address": "10.4.0.2"}     |
| 658561b3-babb-4eb1-90c9-612a39be22ca | ad6ab282599242e6a15796737eae4071 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-f252543c-b2f9-420d-b382-f345e4e76523 | {"subnet_id": "766d53c3-ca8a-4b26-8440-7d09591ff77f", "ip_address": "10.4.0.2"}     |
| 6883bf55-f316-4e07-8611-58c239a28565 | d47543a997284c999a29f4aca3c4a676 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-b816b8fd-66f3-4475-a0bb-325f311bf34d | {"subnet_id": "9c58480e-2be1-45ba-b9d6-056ad7caf1dd", "ip_address": "10.4.0.2"}     |
| 691534fd-08a2-4919-b345-d094049277de | 57b0d5a34a0041e4b129336311f06ad6 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-c069f485-f353-4358-8e3e-694a24bcfd3a | {"subnet_id": "c7b0ec9c-9b4a-48a9-b346-d8a1103ed284", "ip_address": "10.4.0.2"}     |
| 70153715-c172-4079-9629-7263ab855051 | e5e0b98f44de48da975cface2d6589d1 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-4b7b2dd5-e728-4f09-85a7-878a8367e3e1 | {"subnet_id": "27562c7e-8a1c-41fd-bbde-d3f2f914e30b", "ip_address": "10.4.0.3"}     |
| 7a58b159-5790-48ea-8a05-59ad4589d8f4 | 91098fae4b6746fe9f070836919ee700 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-a1826858-2475-45e0-ba31-970e1f2c8c46 | {"subnet_id": "07e31a06-1dba-4a9b-bcc5-3a0023327f46", "ip_address": "10.4.0.2"}     |
| 8791c8b5-5dd5-4579-8479-16574d2e2e57 | ba3dc68fab624332a89f6bbc4bdfa6a7 | network:dhcp | reserved_dhcp_port                                                            | {"subnet_id": "b2149b33-3417-4832-b0a7-f6bd95c77ec0", "ip_address": "10.4.0.2"}     |
| 983e74d3-c573-475d-85c0-db867c2c95dd | b235675844294535a085282854ca20f7 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-a4b0659d-7e38-43f2-bbc7-8f78fac8fbc6 | {"subnet_id": "402d73f2-1d69-4725-93e6-05b227f69174", "ip_address": "10.4.0.3"}     |
| a0104436-fcb4-4546-87d4-9d909596fb1e | c45b3ac84505413cafd1e3a7fe283a99 | network:dhcp | dhcp945e4881-9a43-5e7b-ac70-097af06eb70f-bb0055ce-1277-4f01-8466-08db77329af0 | {"subnet_id": "5f18f878-71c2-45be-b30b-e8f84b287ee9", "ip_address": "192.168.0.2"}  |
| a4dcf6bb-87ea-41f3-ae30-f88d93a14ded | c3485772a56f4126a3b13ec3f8e4e3af | network:dhcp | reserved_dhcp_port                                                            | {"subnet_id": "469e5228-f493-4d87-9a83-6c8ca716501b", "ip_address": "10.4.0.3"}     |
| bc18efe0-b7b7-4a66-b237-4799a8e49c56 | c3485772a56f4126a3b13ec3f8e4e3af | network:dhcp | dhcp945e4881-9a43-5e7b-ac70-097af06eb70f-c271bb15-d6da-4cba-8c24-c563b9e7380a | {"subnet_id": "469e5228-f493-4d87-9a83-6c8ca716501b", "ip_address": "10.4.0.2"}     |
| cae9f6d9-8d6d-482a-8339-921d689b3a46 | ba3dc68fab624332a89f6bbc4bdfa6a7 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-970e4a25-77b2-4946-b908-5a935b4eb714 | {"subnet_id": "b2149b33-3417-4832-b0a7-f6bd95c77ec0", "ip_address": "10.4.0.4"}     |
| e15f43ae-567a-4722-b4b5-a775ac9fbf4c | 7c3d6e7c173b47fe9b022fed8772b62e | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-f3fd446e-2db5-426d-ba75-cbb338a37420 | {"subnet_id": "47fb3526-d496-465e-8096-81381ce7246d", "ip_address": "10.4.0.2"}     |
| e1c7b7e4-872b-4e93-8f6b-10c86e030811 | 787906c03f194361ba0dde2a79c0e6f4 | network:dhcp | dhcp945e4881-9a43-5e7b-ac70-097af06eb70f-ccfc90f7-013f-40e9-abf5-d4738cc9359d | {"subnet_id": "d21a3d3d-c94e-412b-b918-4f3ff3123937", "ip_address": "10.4.0.2"}     |
| f8549792-7183-4d9d-848d-249adf49e5e1 | c45b3ac84505413cafd1e3a7fe283a99 | network:dhcp | dhcp53c914bd-1e13-526a-bad9-54812b015df1-b29b4c1a-10e8-48a9-9ddd-069a3a0ba0bf | {"subnet_id": "8478557d-6c43-4d1c-b0c6-674a3cf02755", "ip_address": "10.4.0.2"}     |
+--------------------------------------+----------------------------------+--------------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+

  • 해결 과정- 컨트롤러 노드

root@controller:/etc/neutron# neutron port-delete 19bd67af-090e-4253-9e4d-52d0bf38489c a4dcf6bb-87ea-41f3-ae30-f88d93a14ded 8791c8b5-5dd5-4579-8479-16574d2e2e57
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Deleted port(s): 19bd67af-090e-4253-9e4d-52d0bf38489c, a4dcf6bb-87ea-41f3-ae30-f88d93a14ded, 8791c8b5-5dd5-4579-8479-16574d2e2e57

# reserved_dhcp_port 삭제 해준다.
  • 해결 과정- 네트워크 노드

root@network:/etc/neutron# ip net |grep dhcp |grep c271bb15-d6da-4cba-8c24-c563b9e7380a
qdhcp-c271bb15-d6da-4cba-8c24-c563b9e7380a (id: 1)
root@network:/etc/neutron# ip net delete qdhcp-c271bb15-d6da-4cba-8c24-c563b9e7380a

root@network:/etc/neutron# service neutron-dhcp-agent restart 

# namespace 삭제후 dhcp agnet 재시작 한다.

Category: 가상화/클라우드

장영호

About the Author ()