메뉴 닫기

resource provider conflict 에러 처리

  • compute node error 
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager [req-9a2e9ab8-59c2-42a9-96d4-3213fe0195f1 – – – – -] Error updating resources for node node.: nova.exception.ResourceProviderCreationFailed: Failed to create resource provider node
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager Traceback (most recent call last):
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/manager.py”, line 9936, in _update_available_resource_for_node
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 896, in update_available_resource
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py”, line 360, in inner
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return f(*args, **kwargs)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 1008, in _update_available_resource
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 1260, in _update
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 49, in wrapped_f
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 206, in call
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 247, in get
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/six.py”, line 703, in reraise
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager raise value
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 200, in call
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 1190, in _update_to_placement
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager prov_tree = self.reportclient.get_provider_tree_and_ensure_root(
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 857, in get_provider_tree_and_ensure_root
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._ensure_resource_provider(
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 644, in _ensure_resource_provider
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager created_rp = self._create_resource_provider(
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 74, in wrapper
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return f(self, *a, **k)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 576, in _create_resource_provider
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager nova.exception.ResourceProviderCreationFailed: Failed to create resource provider node
  • db에서 resource provider uuid, nova uuid 확인 

MariaDB [(none)]> use nova;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nova]> select UUID from compute_nodes where host=’node’;
+————————————–+
| UUID |
+————————————–+
| aaad40c7-1bad-4de8-b5f7-118bae99c712
| 5087e695-0fdb-4f07-b5aa-a60f3e90c1a8 |
| 33acbcc5-be0d-4f1b-9336-1dbec7140961 |
| 5dd29193-aea6-4d80-94a5-0d5d5ac37658 |
+————————————–+

MariaDB [nova]> use placement
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [placement]> select * from resource_providers where name=’node’;
+———————+———————+—-+————————————–+———-+————+——————+——————–+
| created_at | updated_at | id | uuid | name | generation | root_provider_id | parent_provider_id |
+———————+———————+—-+————————————–+———-+————+——————+——————–+
| 2021-07-28 00:45:52 | 2022-11-09 23:42:29 | 21 | 5087e695-0fdb-4f07-b5aa-a60f3e90c1a8 | node | 70 | 21 | NULL |
+———————+———————+—-+————————————–+———-+————+——————+——————–+

  • nova compute_node uuid와 resource_provider uuid 확인 후 uuid reource_provider uuid 수정 

MariaDB [placement]> update resource_providers set uuid=’aaad40c7-1bad-4de8-b5f7-118bae99c712‘ where name=’node’ and uuid=’5087e695-0fdb-4f07-b5aa-a60f3e90c1a8‘;
Query OK, 1 row affected (0.010 sec)
Rows matched: 1 Changed: 1 Warnings: 0

 

MariaDB [placement]> select * from resource_providers where name=’node’;
+———————+———————+—-+————————————–+———-+————+——————+——————–+
| created_at | updated_at | id | uuid | name | generation | root_provider_id | parent_provider_id |
+———————+———————+—-+————————————–+———-+————+——————+——————–+
| 2021-07-28 00:45:52 | 2022-11-09 23:42:29 | 21 | aaad40c7-1bad-4de8-b5f7-118bae99c712  | node | 70 | 21 | NULL |
+———————+———————+—-+————————————–+———-+————+——————+——————–+

  • resource_provider uuid 변경 후 log 확인 ( error x ) -> 정상적으로 서비스 가능 
2022-11-10 02:43:02.899 17232 INFO nova.scheduler.client.report [req-11487eb2-159c-4abc-962c-1d1ec154709a – – – – -] [req-098c889a-a50a-4e2b-99b7-377c18f24d8a] Created resource provider record via placement API for resource provider with UUID 5087e695-0fdb-4f07-b5aa-a60f3e90c1a8 and name node.
2022-11-10 02:43:02.903 17232 INFO nova.virt.libvirt.host [req-11487eb2-159c-4abc-962c-1d1ec154709a – – – – -] kernel doesn’t support AMD SEV
  •  resource_provider uuid 값을 수정하고도 아래와 같은 error 발생 
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager Traceback (most recent call last):
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/manager.py”, line 9936, in _update_available_resource_for_node
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 896, in update_available_resource
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py”, line 360, in inner
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return f(*args, **kwargs)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 1008, in _update_available_resource
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 1260, in _update
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 49, in wrapped_f
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 206, in call
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 247, in get
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/six.py”, line 703, in reraise
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager raise value
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/retrying.py”, line 200, in call
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py”, line 1190, in _update_to_placement
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager prov_tree = self.reportclient.get_provider_tree_and_ensure_root(
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 857, in get_provider_tree_and_ensure_root
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager self._ensure_resource_provider(
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 644, in _ensure_resource_provider
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager created_rp = self._create_resource_provider(
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 74, in wrapper
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager return f(self, *a, **k)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager File “/usr/lib/python3/dist-packages/nova/scheduler/client/report.py”, line 576, in _create_resource_provider
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager nova.exception.ResourceProviderCreationFailed: Failed to create resource provider node
2022-11-10 02:42:29.070 17102 ERROR nova.compute.manager
  • node에 할당되어있는 oid 확인

root@con:~# token=`openstack token issue -f value -c id`

root@con:~# echo $token
gAAAAABgNd_fLcg40DWpTUUUzulOeqwP9Ov6vmjZu0SsuP2FUV9hoaZYLphI1xImFn0ZhtnK03ZO0xQfJdZXGY75ULwlol-6EyL0yzqDGo_EJN5LnxIpmik_eJuPBSUSLx0ow-3hV9mOg0z42GgklXHFN1xKdxx3EPY-4M3ksPyn3OZPvOhbVLY

root@con:~# curl http://controller:port/resource_providers -H “x-auth-token: $token” | python -m json.tool  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21149  100 21149    0     0   102k      0 –:–:– –:–:– –:–:–  102k
{
    “resource_providers”: [
        {
..
..
..
        {
            “generation”: 5,
            “links”: [
                {
                    “href”: “/resource_providers/0a7ec534-50b4-42c6-a832-6a350cae5f7e”,
                    “rel”: “self”
                },
                {
                    “href”: “/resource_providers/0a7ec534-50b4-42c6-a832-6a350cae5f7e/inventories”,
                    “rel”: “inventories”
                },
                {
                    “href”: “/resource_providers/0a7ec534-50b4-42c6-a832-6a350cae5f7e/usages”,
                    “rel”: “usages”
                }
            ],
            “name”: “node”,
           “uuid”: “0a7ec534-50b4-42c6-a832-6a350cae5f7e”
        }
..
..

  • 해당 oid 삭제 

root@con:~# curl http://controller:port/resource_providers/0a7ec534-50b4-42c6-a832-6a350cae5f7e -H “x-auth-token: $token” -X DELETE

root@con:~# curl http://controller:port/resource_providers -H “x-auth-token: $token” | python -m json.tool | grep node
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20767 100 20767 0 0 533k 0 –:–:– –:–:– –:–:– 533k

  • node에 해당되어있는 instances 실제 host(node-2)의 id로 변경 (0a7ec534-50b4-42c6-a832-6a350cae5f7e uuid의 host확인 후 작업)

MariaDB [placement]> select * from resource_providers where name=’node-2′;
+———————+———————+——–+————————————–+———-+————+——————+——————–+
| created_at | updated_at | id | uuid | name | generation | root_provider_id | parent_provider_id |
+———————+———————+——–+————————————–+———-+————+——————+——————–+
| 2022-11-03 23:51:40 | 2022-11-06 10:25:34 | 337179 | 3bf5b37e-3317-4389-9ec0-abc5ce89b8bd | node | 10 | 337179 | NULL |
+———————+———————+——–+————————————–+———-+————+——————+——————–+

MariaDB [placement]> select * from resource_providers where name=’node’;
+———————+———————+—-+————————————–+———-+————+——————+——————–+
| created_at | updated_at | id | uuid | name | generation | root_provider_id | parent_provider_id |
+———————+———————+—-+————————————–+———-+————+——————+——————–+
| 2021-07-28 00:45:52 | 2022-11-09 23:42:29 | 21 | aaad40c7-1bad-4de8-b5f7-118bae99c712 | node | 70 | 21 | NULL |
+———————+———————+—-+————————————–+———-+————+——————+——————–+

MariaDB [placement]> select * from allocations where consumer_id=’0a7ec534-50b4-42c6-a832-6a350cae5f7e’;
+———————+————+——+———————-+————————————–+——————-+——+
| created_at | updated_at | id | resource_provider_id | consumer_id | resource_class_id | used |
+———————+————+——+———————-+————————————–+——————-+——+
| 2022-02-08 06:29:46 | NULL | 5863 | 21 | 0a7ec534-50b4-42c6-a832-6a350cae5f7e | 0 | 1 |
| 2022-02-08 06:29:46 | NULL | 5864 | 21 | 0a7ec534-50b4-42c6-a832-6a350cae5f7e | 1 | 2048 |
| 2022-02-08 06:29:46 | NULL | 5865 | 21 | 0a7ec534-50b4-42c6-a832-6a350cae5f7e | 2 | 25 |
+———————+————+——+———————-+————————————–+——————-+——+

MariaDB [placement]> update allocations set resource_provider_id=’337179′ where consumer_id=’0a7ec534-50b4-42c6-a832-6a350cae5f7e’;
Query OK, 3 rows affected (0.00 sec)
Rows matched: 3 Changed: 3 Warnings:

MariaDB [nova_api]> select * from allocations where consumer_id=’0a7ec534-50b4-42c6-a832-6a350cae5f7e’;
+———————+————+——+———————-+————————————–+——————-+——+
| created_at | updated_at | id | resource_provider_id | consumer_id | resource_class_id | used |
+———————+————+——+———————-+————————————–+——————-+——+
| 2022-02-08 06:29:46 | NULL | 5863 | 337179 | 0a7ec534-50b4-42c6-a832-6a350cae5f7e | 0 | 1 |
| 2022-02-08 06:29:46 | NULL | 5864 | 337179 | 0a7ec534-50b4-42c6-a832-6a350cae5f7e | 1 | 2048 |
| 2022-02-08 06:29:46 | NULL | 5865 | 337179 | 0a7ec534-50b4-42c6-a832-6a350cae5f7e | 2 | 25 |
+———————+————+——+———————-+————————————–+——————-+——+

  • 수정 후 서비스 재시작 

root@node:~# systemctl start nova-compute.service neutron-l3-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service

2022-11-10 02:43:02.899 17232 INFO nova.scheduler.client.report [req-11487eb2-159c-4abc-962c-1d1ec154709a – – – – -] [req-098c889a-a50a-4e2b-99b7-377c18f24d8a] Created resource provider record via placement API for resource provider with UUID 5087e695-0fdb-4f07-b5aa-a60f3e90c1a8 and name node.
2022-11-10 02:43:02.903 17232 INFO nova.virt.libvirt.host [req-11487eb2-159c-4abc-962c-1d1ec154709a – – – – -] kernel doesn’t support AMD SEV

 

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x