We're updating the issue view to help you get more done. 

Cannot add new node to cluster after failover


After a cluster start, the encryption key (and other stuff) is stored in /etc/cloudify/cluster on the manager you use for the `cfy cluster start` command. The other managers don't have this data so if they become master later, new nodes can't join the cluster.

Steps to reproduce:
1) Bootstrap 3 managers
2) Start cluster on manager1
cfy profiles use <node1_ip>
cfy cluster start --cluster-node-name manager1

3) Join cluster on manager2
cfy profiles use <node2_ip>
cfy cluster join <node1_ip> --cluster-node-name manager2

4) Set manager2 as active
cfy cluster set-active manager2

5) Join manager3 to cluster via manager2
cfy profiles use <node3_ip>
cfy cluster join <node2_ip> --cluster-node-name manager3

Step 5 fails with the following error on manager2:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 15/09/2017 18:10:19 [ERROR] [manager_rest.server] Internal Error Traceback (most recent call last): File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/opt/manager/env/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/opt/manager/env/lib/python2.7/site-packages/flask_restful/__init__.py", line 261, in wrapper resp = resource(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/flask/views.py", line 84, in view return self.dispatch_request(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/flask_restful/__init__.py", line 341, in dispatch_request resp = meth(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/cloudify_premium/ha/web.py", line 48, in inner return f(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/security/secured_resource.py", line 38, in wrapper return func(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/rest_decorators.py", line 95, in wrapper return func(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/rest_decorators.py", line 135, in wrapper response = f(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/manager_rest/rest/resources_v3.py", line 427, in put return cluster.add_node(details) File "/opt/manager/env/lib/python2.7/site-packages/cloudify_premium/ha/controller.py", line 71, in _wrapper return f(*args, **kwargs) File "/opt/manager/env/lib/python2.7/site-packages/cloudify_premium/ha/controller.py", line 248, in add_node 'encryption_key': node_status['encryption_key'] File "/opt/manager/env/lib/python2.7/site-packages/cloudify_premium/ha/utils.py", line 105, in __getitem__ return self._load()[key] KeyError: 'encryption_key'





Kieran Spear




Bug Type


Target Version




Fix versions

Affects versions