Plugin Install Fails intermittently

Description

Trying to run a blueprint (virtual machine example on GCP).

Plugins all exist on manager.

During deployment creation, fails with:

2020-07-15 08:14:36.963 CFY <vm> Task failed 'cloudify_agent.operations.install_plugins' -> Timeout waiting for plugin to be installed. Plugin info: [{u'distribution_release': u'core', u'archive_name': u'installing-cloudify_utilities_plugin-1.23.5-centos-Core-py27.py36-none-linux_x86_64.wgn', u'package_name': u'cloudify-utilities-plugin', u'distribution_version': u'7.8.2003', u'tenant_name': u'default_tenant', u'package_version': u'1.23.5', u'distribution': u'centos', u'visibility': u'tenant', u'file_server_path': u'', u'resource_availability': u'tenant', u'package_source': u'.', u'id': u'feaffed8-c240-4ed4-9a83-c44e0624ce17', u'yaml_url_path': u'plugin:cloudify-utilities-plugin?version=1.23.5&distribution=centos', u'supported_py_versions': [u'py27', u'py36'], u'uploaded_at': u'2020-07-15T07:56:28.561Z', u'created_by': u'admin', u'private_resource': False, u'supported_platform': u'linux_x86_64', u'wheels': [u'ipaddress-1.0.23-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl', u'six-1.15.0-py2.py3-none-any.whl', u'decorator-4.4.2-py2.py3-none-any.whl', u'smmap-3.0.4-py2.py3-none-any.whl', u'ply-3.11-py2.py3-none-any.whl', u'pycparser-2.20-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py3-none-any.whl', u'PyYAML-3.10-cp27-cp27mu-linux_x86_64.whl', u'cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb-4.0.5-py3-none-any.whl', u'PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl', u'idna-2.9-py2.py3-none-any.whl', u'xmltodict-0.12.0-py2.py3-none-any.whl', u'paramiko-2.7.1-py2.py3-none-any.whl', u'retrying-1.3.3-py2-none-any.whl', u'pysmi-0.3.4-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp27-cp27mu-manylinux1_x86_64.whl', u'bottle-0.12.7-py2-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py2-none-any.whl', u'retrying-1.3.3-py3-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py3-none-any.whl', u'pycdlib-1.10.0-py2.py3-none-any.whl', u'pysnmp-4.4.5-py2.py3-none-any.whl', u'GitPython-3.1.3-py3-none-any.whl', u'MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl', u'pika-0.11.2-py2.py3-none-any.whl', u'bottle-0.12.7-py3-none-any.whl', u'chardet-3.0.4-py2.py3-none-any.whl', u'pyasn1-0.4.8-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py2-none-any.whl', u'proxy_tools-0.1.0-py2-none-any.whl', u'proxy_tools-0.1.0-py3-none-any.whl', u'GitPython-2.1.15-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp27-cp27mu-manylinux1_x86_64.whl', u'MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl', u'ruamel.yaml.clib-0.2.0-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.4-py3-none-any.whl', u'PyNaCl-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl', u'ruamel.yaml-0.16.10-py2.py3-none-any.whl', u'requests-2.24.0-py2.py3-none-any.whl', u'gitdb-0.6.4-cp27-cp27mu-linux_x86_64.whl', u'smmap2-3.0.1-py2-none-any.whl', u'networkx-1.9.1-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py3-none-any.whl', u'cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl', u'enum34-1.1.10-py2-none-any.whl', u'ruamel.ordereddict-0.4.14-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.2-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp36-cp36m-manylinux1_x86_64.whl', u'cffi-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl', u'Jinja2-2.10-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py2-none-any.whl', u'PyYAML-3.10-cp36-cp36m-linux_x86_64.whl', u'cryptography-2.9.2-cp27-cp27mu-manylinux2010_x86_64.whl', u'certifi-2020.6.20-py2.py3-none-any.whl', u'urllib3-1.25.9-py2.py3-none-any.whl', u'requests_toolbelt-0.8.0-py2.py3-none-any.whl', u'ruamel.yaml.clib-0.2.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb2-2.0.6-py2.py3-none-any.whl'], u'excluded_wheels': None}]
Traceback (most recent call last):
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks.py", line 726, in local_task_wrapper
result = self.local_task(**self.kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 804, in dispatch
return handler.handle_or_dispatch_to_subprocess_if_remote()
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 194, in handle_or_dispatch_to_subprocess_if_remote
return self.handle()
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 465, in handle
result = self._run_operation_func(ctx, kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 529, in _run_operation_func
return self.func(*self.args, **kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_agent/operations.py", line 63, in install_plugins
blueprint_id=ctx.blueprint.id)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_agent/api/plugins/installer.py", line 113, in install
tmp_plugin_dir=tmp_plugin_dir)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_agent/api/plugins/installer.py", line 157, in _install_managed_plugin
'Plugin info: [{0}] '.format(managed_plugin))
NonRecoverableError: Timeout waiting for plugin to be installed. Plugin info: [{u'distribution_release': u'core', u'archive_name': u'installing-cloudify_utilities_plugin-1.23.5-centos-Core-py27.py36-none-linux_x86_64.wgn', u'package_name': u'cloudify-utilities-plugin', u'distribution_version': u'7.8.2003', u'tenant_name': u'default_tenant', u'package_version': u'1.23.5', u'distribution': u'centos', u'visibility': u'tenant', u'file_server_path': u'', u'resource_availability': u'tenant', u'package_source': u'.', u'id': u'feaffed8-c240-4ed4-9a83-c44e0624ce17', u'yaml_url_path': u'plugin:cloudify-utilities-plugin?version=1.23.5&distribution=centos', u'supported_py_versions': [u'py27', u'py36'], u'uploaded_at': u'2020-07-15T07:56:28.561Z', u'created_by': u'admin', u'private_resource': False, u'supported_platform': u'linux_x86_64', u'wheels': [u'ipaddress-1.0.23-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl', u'six-1.15.0-py2.py3-none-any.whl', u'decorator-4.4.2-py2.py3-none-any.whl', u'smmap-3.0.4-py2.py3-none-any.whl', u'ply-3.11-py2.py3-none-any.whl', u'pycparser-2.20-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py3-none-any.whl', u'PyYAML-3.10-cp27-cp27mu-linux_x86_64.whl', u'cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb-4.0.5-py3-none-any.whl', u'PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl', u'idna-2.9-py2.py3-none-any.whl', u'xmltodict-0.12.0-py2.py3-none-any.whl', u'paramiko-2.7.1-py2.py3-none-any.whl', u'retrying-1.3.3-py2-none-any.whl', u'pysmi-0.3.4-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp27-cp27mu-manylinux1_x86_64.whl', u'bottle-0.12.7-py2-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py2-none-any.whl', u'retrying-1.3.3-py3-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py3-none-any.whl', u'pycdlib-1.10.0-py2.py3-none-any.whl', u'pysnmp-4.4.5-py2.py3-none-any.whl', u'GitPython-3.1.3-py3-none-any.whl', u'MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl', u'pika-0.11.2-py2.py3-none-any.whl', u'bottle-0.12.7-py3-none-any.whl', u'chardet-3.0.4-py2.py3-none-any.whl', u'pyasn1-0.4.8-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py2-none-any.whl', u'proxy_tools-0.1.0-py2-none-any.whl', u'proxy_tools-0.1.0-py3-none-any.whl', u'GitPython-2.1.15-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp27-cp27mu-manylinux1_x86_64.whl', u'MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl', u'ruamel.yaml.clib-0.2.0-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.4-py3-none-any.whl', u'PyNaCl-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl', u'ruamel.yaml-0.16.10-py2.py3-none-any.whl', u'requests-2.24.0-py2.py3-none-any.whl', u'gitdb-0.6.4-cp27-cp27mu-linux_x86_64.whl', u'smmap2-3.0.1-py2-none-any.whl', u'networkx-1.9.1-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py3-none-any.whl', u'cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl', u'enum34-1.1.10-py2-none-any.whl', u'ruamel.ordereddict-0.4.14-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.2-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp36-cp36m-manylinux1_x86_64.whl', u'cffi-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl', u'Jinja2-2.10-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py2-none-any.whl', u'PyYAML-3.10-cp36-cp36m-linux_x86_64.whl', u'cryptography-2.9.2-cp27-cp27mu-manylinux2010_x86_64.whl', u'certifi-2020.6.20-py2.py3-none-any.whl', u'urllib3-1.25.9-py2.py3-none-any.whl', u'requests_toolbelt-0.8.0-py2.py3-none-any.whl', u'ruamel.yaml.clib-0.2.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb2-2.0.6-py2.py3-none-any.whl'], u'excluded_wheels': None}]

There is a high risk of users encountering this issue regularly. I have experienced it two to three times in the last few days.

Steps to Reproduce

Environment:
OS (CLI), HA cluster, cloud provider
------------------------------------

Steps to reproduce:
------------------
1.
2.
3.

Expected result:
---------------

Actual result:
-------------

Why Propose Close?

None

Activity

Show:
Alex Molev
July 22, 2020, 12:24 PM

I guess it’s related to lazy plugin install, can we wrap it with a pretty message

Assignee

Łukasz Maksymczuk

Reporter

Trammell -

Severity

Critical

Target Version

5.1

Premium Only

no

Found In Version

5.1

Bug Type

unknown

Customer Encountered

No

Release Notes

no

Priority

High