Error when removing a RabbitMQ cluster node

Description

I got an error when I tried to remove a Rabbit node from a cluster.
When I ran: `cfy cluster brokers remove rabbitmq_3`, this was the error:

26/11/2019 14:19:37 [ERROR] [flask.app] Exception on /api/v3.1/brokers/rabbitmq_3 [DELETE]
Traceback (most recent call last):
File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/manager/env/lib/python2.7/site-packages/flask_restful/_init_.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/opt/manager/env/lib/python2.7/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/opt/manager/env/lib/python2.7/site-packages/flask_restful/_init_.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/security/secured_resource.py", line 57, in wrapper
response = func(*args, **kwargs)
File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/security/authorization.py", line 63, in wrapper
return func(*args, **kwargs)
File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/rest_decorators.py", line 122, in wrapper
response = f(*args, **kwargs)
File "/opt/manager/env/lib/python2.7/site-packages/cloudify_premium/manager.py", line 198, in delete
update_agents(sm)
File "/opt/manager/env/lib/python2.7/site-packages/cloudify_premium/ha/agents.py", line 75, in update_agents
with amqp_client:
File "/opt/manager/env/lib/python2.7/site-packages/cloudify/amqp_client.py", line 238, in _enter_
self.consume_in_thread()
File "/opt/manager/env/lib/python2.7/site-packages/cloudify/amqp_client.py", line 234, in consume_in_thread
raise self._error
ConnectionClosed: Connection to 172.16.4.45:5671 failed: [Errno 113] No route to host
26/11/2019 14:19:40 [ERROR] [flask.app] User unauthorized: No authentication info provided
26/11/2019 14:19:41 [ERROR] [flask.app] Broker check failed with <class 'pika.exceptions.ConnectionClosed'>: Connection to 172.16.4.45:5671 failed: [Errno 113] No route to host

Steps to Reproduce

Environment:
OS (CLI), HA cluster, cloud provider
------------------------------------
Cloudify Cluster with 3 managers, 3 db nodes, 3 rabbit.

Steps to reproduce:
------------------
1. Run the cluster script
2. Shut off one of the rabbit nodes (I chose rabbit3)
3. Run cfy_manager brokers remove -r rabbitmq_3
4. Run cfy cluster brokers remove rabbitmq_3

Expected result:
---------------
Successfully removed rabbit node.

Actual result:
-------------
Failure when trying to update the agents.

Why Propose Close?

None

Activity

Show:
David Ginzbourg
December 4, 2019, 4:16 PM

Will try to replicate this tomorrow.

Assignee

Michael Glokhman

Reporter

Inbal Amrani

Labels

None

Severity

High

Target Version

5.0.5

Premium Only

yes

Found In Version

5.0

QA Owner

None

Bug Type

regression bug

Customer Encountered

No

Customer Name

None

Release Notes

no

Priority

None

Epic Link

Sprint

None

Priority

Blocker
Configure