Agents upgrade doesn't work on snapshot is taken from HA cluster

Description

Agents upgrade doesn't work after fail-over in HA cluster.
According to runtime property it tries to connect to the origin master manager in cluster.

Steps to Reproduce

Environment:
OS (CLI), HA cluster, cloud provider
------------------------------------

Steps to reproduce:
------------------
1. Create HA cluster of 2 managers (m1, m2)
2. Upload blueprint, create deployment and install deployment
3. Create snapshot
4. Download and upload snapshot to the new manager (m3)
5. Restore snapshot
6. Install agents for existing deployments

Expected result:
---------------
Agents have been installed

Actual result:
-------------
Agents install has failed.

Running workflow 'install_new_agents' for all installed deployments
2018-04-17 07:22:32.060 CFY <hello-simple> Starting 'install_new_agents' workflow execution
2018-04-17 07:22:33.086 CFY <hello-simple> [host_g61m2i] Validating agent connection.
2018-04-17 07:22:33.086 CFY <hello-simple> [host_g61m2i.validate_amqp] Sending task 'cloudify_agent.operations.validate_agent_amqp'
2018-04-17 07:22:33.086 CFY <hello-simple> [host_g61m2i.validate_amqp] Task started 'cloudify_agent.operations.validate_agent_amqp'
2018-04-17 07:22:33.969 LOG <hello-simple> [host_g61m2i.validate_amqp] INFO: Trying old AMQP...
2018-04-17 07:22:34.156 LOG <hello-simple> [host_g61m2i.validate_amqp] INFO: Connecting to amqp://rabbitmq_user_tenantPLL8luR6AJcd2cm4Wuc8j9I6kJ7dQhz@10.0.0.7:5671/rabbitmq_vhost_tenant
2018-04-17 07:22:34.452 CFY <hello-simple> [host_g61m2i.validate_amqp] Task succeeded 'cloudify_agent.operations.validate_agent_amqp'
2018-04-17 07:22:34.452 CFY <hello-simple> [host_g61m2i.validate_amqp] Task succeeded 'cloudify_agent.operations.validate_agent_amqp'
2018-04-17 07:22:35.119 CFY <hello-simple> [host_g61m2i] Validation done
2018-04-17 07:22:35.119 CFY <hello-simple> [host_g61m2i] Installing new agent
2018-04-17 07:22:35.119 CFY <hello-simple> [host_g61m2i.create_amqp] Sending task 'cloudify_agent.operations.create_agent_amqp'
2018-04-17 07:22:35.119 CFY <hello-simple> [host_g61m2i.create_amqp] Task started 'cloudify_agent.operations.create_agent_amqp'
2018-04-17 07:22:36.189 LOG <hello-simple> [host_g61m2i.create_amqp] INFO: Connecting to amqp://rabbitmq_user_tenantPLL8luR6AJcd2cm4Wuc8j9I6kJ7dQhz@10.0.0.7:5671/rabbitmq_vhost_tenant
2018-04-17 07:22:37.301 CFY <hello-simple> [host_g61m2i.create_amqp] Task failed 'cloudify_agent.operations.create_agent_amqp' -> host must be a str or unicode str
2018-04-17 07:22:52.470 CFY <hello-simple> [host_g61m2i.create_amqp] Sending task 'cloudify_agent.operations.create_agent_amqp' [retry 1/60]
2018-04-17 07:22:53.144 CFY <hello-simple> [host_g61m2i.create_amqp] Task started 'cloudify_agent.operations.create_agent_amqp' [retry 1/60]
2018-04-17 07:22:53.618 LOG <hello-simple> [host_g61m2i.create_amqp] INFO: Connecting to amqp://rabbitmq_user_tenantPLL8luR6AJcd2cm4Wuc8j9I6kJ7dQhz@10.0.0.7:5671/rabbitmq_vhost_tenant
2018-04-17 07:23:24.101 CFY <hello-simple> [host_g61m2i.create_amqp] Task failed 'cloudify_agent.operations.create_agent_amqp' -> Could not access tasks list for agent host_g61m2i_49a858b5-28d3-40f1-a6a4-cd02af00c876 [retry 1/60]
2018-04-17 07:23:25.159 CFY <hello-simple> 'install_new_agents' workflow execution failed: RuntimeError: Workflow failed: Task failed 'cloudify_agent.operations.create_agent_amqp' -> Could not access tasks list for agent host_g61m2i_49a858b5-28d3-40f1-a6a4-cd02af00c876
Summary:
Execution of workflow 'install_new_agents' for deployment 'hello-simple' failed. [error=Traceback (most recent call last):
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 519, in _remote_workflow_child_thread
workflow_result = self._execute_workflow_function()
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 550, in _execute_workflow_function
result = self.func(*self.args, **self.kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/workflows.py", line 324, in install_new_agents
graph.execute()
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 133, in execute
self._handle_terminated_task(task)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 207, in _handle_terminated_task
raise RuntimeError(message)
RuntimeError: Workflow failed: Task failed 'cloudify_agent.operations.create_agent_amqp' -> Could not access tasks list for agent host_g61m2i_49a858b5-28d3-40f1-a6a4-cd02af00c876
]

Why Propose Close?

None

Status

Assignee

Łukasz Maksymczuk

Reporter

Dmitry Gertsman

Labels

None

Severity

Critical

Target Version

4.4

Premium Only

no

Found In Version

4.4

QA Owner

Uri Wygodny

Bug Type

legacy bug

Customer Encountered

Yes

Customer Name

None

Release Notes

yes

Priority

High

Sprint

None

Priority

Unprioritized
Configure