Bootstrap fails during in-place upgrade process after manager teardown in HA cluster

Description

Prerequisites:

  • Cloudify premium CLI edition installed v 4.1.1

  • 2 * Cloudify manager v 4.1.1bootstrapped and running.

Steps:

  1. Install cli on the first vm- sudo rpm -I http://gigaspaces-repository-eu.s3.amazonaws.com/cloudify/4.1.1/ga-release/cloudify-4.1.1~ga.el6.x86_64.rpm

  2. Edit the inputs credentials and bootstrap the manager on the second vm- cfy bootstrap <MANAGER BLUEPRINT YAML FILE> -i <MANAGER BLUEPRINT INPUT FILE>

  3. Verify manager's version- cfy --version

  4. Edit the inputs credentials and bootstrap the manager on the third vm- cfy bootstrap <MANAGER BLUEPRINT YAML FILE> -i <MANAGER BLUEPRINT INPUT FILE>

  5. Verify manager's version- cfy --version

  6. Connect to the first manager (second vm)- cfy profiles use <PUBLIC_IP>

  7. Start cluster- cfy cluster start —cluster-host-ip<PUBLIC_IP> —cluster-node-name<CLUSTER-NAME>

  8. Connect to the second manager (third vm)- cfy profiles use <PUBLIC_IP>

  9. Join cluster- cfy cluster join <PUBLIC_IP (FIRST MANAGER)> —cluster-host-ip<PUBLIC_IP> —cluster-node-name<CLUSTER_NAME>

  10. Remove the “leader” manager (first manager- second vm)- cfy cluster node remove <FIRST_MANAGER_NODE_NAME>

  11. Connect to the second manager (third vm)- cfy profiles use<PUBLIC_IP>

  12. Verify the second manager is now the "leader"- cfy cluster node list

  13. Re-create a 4.2 manager on the same ip (on the third vm- second manager):
    Teardown- Cfy teardown -f —ignore-deployments
    Uninstall cli pkg- Sudo rpm -e <pkg>

  14. Edit the inputs credentials and bootstrap the manager on the second vm- cfy bootstrap <MANAGER BLUEPRINT YAML FILE> -i <MANAGER BLUEPRINT INPUT FILE>

Expected result:
Bootstrap completed during in-place upgrade process after manager teardown in HA cluster

Actually result:
Bootstrap fails during in-place upgrade process after manager teardown in HA cluster

[10.239.1.36] out: /tmp2017-11-19 09:43:42.748 LOG <manager> [python_runtime_7550jy.create] INFO: Installing Python Requirements...
2017-11-19 09:43:42.852 LOG <manager> [python_runtime_7550jy.create] INFO: Checking whether SELinux in enforced...
2017-11-19 09:43:43.175 LOG <manager> [python_runtime_7550jy.create] INFO: Checking whether /opt/cloudify/python/resources/python-pip-7.1.0-1.el7.noarch.rpm is already installed...
2017-11-19 09:43:43.715 CFY <manager> [python_runtime_7550jy.create] Task succeeded 'fabric_plugin.tasks.run_script'
2017-11-19 09:43:43.715 CFY <manager> [consul_g5fy8f.create] Task started 'fabric_plugin.tasks.run_script'
[10.239.1.36] out: /tmp[10.239.1.36] out: Traceback (most recent call last):
[10.239.1.36] out: File "/tmp/cloudify-ctx/scripts/tmphfj93q-create.py-MSEGBS7G", line 48, in <module>
[10.239.1.36] out: install_consul()
[10.239.1.36] out: File "/tmp/cloudify-ctx/scripts/tmphfj93q-create.py-MSEGBS7G", line 42, in install_consul
[10.239.1.36] out: utils.move(join(temp_dir, 'consul'), consul_binary)
[10.239.1.36] out: File "/tmp/cloudify-ctx/scripts/utils.py", line 261, in move
[10.239.1.36] out: copy(source, destination)
[10.239.1.36] out: File "/tmp/cloudify-ctx/scripts/utils.py", line 272, in copy
[10.239.1.36] out: sudo(['cp', '-rp', source, destination])
[10.239.1.36] out: File "/tmp/cloudify-ctx/scripts/utils.py", line 148, in sudo
[10.239.1.36] out: return run(command=command, *args, **kwargs)
[10.239.1.36] out: File "/tmp/cloudify-ctx/scripts/utils.py", line 137, in run
[10.239.1.36] out: raise RuntimeError(msg)
[10.239.1.36] out: RuntimeError: Failed running command: sudo cp -rp /tmp/tmpbiD4Ba/consul /opt/consul/consul (cp: cannot create regular file ‘/opt/consul/consul’: Text file busy
[10.239.1.36] out: ).
[10.239.1.36] out:

Fatal error: run() received nonzero return code 1 while executing!

WORKAROUND:
Run the 4.1.0 teardown using a custom teardown script:
http://repository.cloudifysource.org/cloudify/4.2.0/ga-release/cfy_teardown_4_1_0_and_higher.sh

Status

Assignee

Yariv Hamamy

Reporter

Lital Hamami

Labels

Severity

None

Bug Type

None

Target Version

None

Severity

None

Epic Link

Sprint

None

Fix versions

Affects versions

4.2