Plugin Install Fails intermittently

Description

Trying to run a blueprint (virtual machine example on GCP).

Plugins all exist on manager.

During deployment creation, fails with:

2020-07-15 08:14:36.963 CFY <vm> Task failed 'cloudify_agent.operations.install_plugins' -> Timeout waiting for plugin to be installed. Plugin info: [{u'distribution_release': u'core', u'archive_name': u'installing-cloudify_utilities_plugin-1.23.5-centos-Core-py27.py36-none-linux_x86_64.wgn', u'package_name': u'cloudify-utilities-plugin', u'distribution_version': u'7.8.2003', u'tenant_name': u'default_tenant', u'package_version': u'1.23.5', u'distribution': u'centos', u'visibility': u'tenant', u'file_server_path': u'', u'resource_availability': u'tenant', u'package_source': u'.', u'id': u'feaffed8-c240-4ed4-9a83-c44e0624ce17', u'yaml_url_path': u'plugin:cloudify-utilities-plugin?version=1.23.5&distribution=centos', u'supported_py_versions': [u'py27', u'py36'], u'uploaded_at': u'2020-07-15T07:56:28.561Z', u'created_by': u'admin', u'private_resource': False, u'supported_platform': u'linux_x86_64', u'wheels': [u'ipaddress-1.0.23-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl', u'six-1.15.0-py2.py3-none-any.whl', u'decorator-4.4.2-py2.py3-none-any.whl', u'smmap-3.0.4-py2.py3-none-any.whl', u'ply-3.11-py2.py3-none-any.whl', u'pycparser-2.20-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py3-none-any.whl', u'PyYAML-3.10-cp27-cp27mu-linux_x86_64.whl', u'cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb-4.0.5-py3-none-any.whl', u'PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl', u'idna-2.9-py2.py3-none-any.whl', u'xmltodict-0.12.0-py2.py3-none-any.whl', u'paramiko-2.7.1-py2.py3-none-any.whl', u'retrying-1.3.3-py2-none-any.whl', u'pysmi-0.3.4-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp27-cp27mu-manylinux1_x86_64.whl', u'bottle-0.12.7-py2-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py2-none-any.whl', u'retrying-1.3.3-py3-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py3-none-any.whl', u'pycdlib-1.10.0-py2.py3-none-any.whl', u'pysnmp-4.4.5-py2.py3-none-any.whl', u'GitPython-3.1.3-py3-none-any.whl', u'MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl', u'pika-0.11.2-py2.py3-none-any.whl', u'bottle-0.12.7-py3-none-any.whl', u'chardet-3.0.4-py2.py3-none-any.whl', u'pyasn1-0.4.8-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py2-none-any.whl', u'proxy_tools-0.1.0-py2-none-any.whl', u'proxy_tools-0.1.0-py3-none-any.whl', u'GitPython-2.1.15-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp27-cp27mu-manylinux1_x86_64.whl', u'MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl', u'ruamel.yaml.clib-0.2.0-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.4-py3-none-any.whl', u'PyNaCl-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl', u'ruamel.yaml-0.16.10-py2.py3-none-any.whl', u'requests-2.24.0-py2.py3-none-any.whl', u'gitdb-0.6.4-cp27-cp27mu-linux_x86_64.whl', u'smmap2-3.0.1-py2-none-any.whl', u'networkx-1.9.1-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py3-none-any.whl', u'cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl', u'enum34-1.1.10-py2-none-any.whl', u'ruamel.ordereddict-0.4.14-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.2-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp36-cp36m-manylinux1_x86_64.whl', u'cffi-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl', u'Jinja2-2.10-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py2-none-any.whl', u'PyYAML-3.10-cp36-cp36m-linux_x86_64.whl', u'cryptography-2.9.2-cp27-cp27mu-manylinux2010_x86_64.whl', u'certifi-2020.6.20-py2.py3-none-any.whl', u'urllib3-1.25.9-py2.py3-none-any.whl', u'requests_toolbelt-0.8.0-py2.py3-none-any.whl', u'ruamel.yaml.clib-0.2.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb2-2.0.6-py2.py3-none-any.whl'], u'excluded_wheels': None}]
Traceback (most recent call last):
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks.py", line 726, in local_task_wrapper
result = self.local_task(**self.kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 804, in dispatch
return handler.handle_or_dispatch_to_subprocess_if_remote()
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 194, in handle_or_dispatch_to_subprocess_if_remote
return self.handle()
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 465, in handle
result = self._run_operation_func(ctx, kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/dispatch.py", line 529, in _run_operation_func
return self.func(*self.args, **kwargs)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_agent/operations.py", line 63, in install_plugins
blueprint_id=ctx.blueprint.id)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_agent/api/plugins/installer.py", line 113, in install
tmp_plugin_dir=tmp_plugin_dir)
File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify_agent/api/plugins/installer.py", line 157, in _install_managed_plugin
'Plugin info: [{0}] '.format(managed_plugin))
NonRecoverableError: Timeout waiting for plugin to be installed. Plugin info: [{u'distribution_release': u'core', u'archive_name': u'installing-cloudify_utilities_plugin-1.23.5-centos-Core-py27.py36-none-linux_x86_64.wgn', u'package_name': u'cloudify-utilities-plugin', u'distribution_version': u'7.8.2003', u'tenant_name': u'default_tenant', u'package_version': u'1.23.5', u'distribution': u'centos', u'visibility': u'tenant', u'file_server_path': u'', u'resource_availability': u'tenant', u'package_source': u'.', u'id': u'feaffed8-c240-4ed4-9a83-c44e0624ce17', u'yaml_url_path': u'plugin:cloudify-utilities-plugin?version=1.23.5&distribution=centos', u'supported_py_versions': [u'py27', u'py36'], u'uploaded_at': u'2020-07-15T07:56:28.561Z', u'created_by': u'admin', u'private_resource': False, u'supported_platform': u'linux_x86_64', u'wheels': [u'ipaddress-1.0.23-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl', u'six-1.15.0-py2.py3-none-any.whl', u'decorator-4.4.2-py2.py3-none-any.whl', u'smmap-3.0.4-py2.py3-none-any.whl', u'ply-3.11-py2.py3-none-any.whl', u'pycparser-2.20-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py3-none-any.whl', u'PyYAML-3.10-cp27-cp27mu-linux_x86_64.whl', u'cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb-4.0.5-py3-none-any.whl', u'PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl', u'idna-2.9-py2.py3-none-any.whl', u'xmltodict-0.12.0-py2.py3-none-any.whl', u'paramiko-2.7.1-py2.py3-none-any.whl', u'retrying-1.3.3-py2-none-any.whl', u'pysmi-0.3.4-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp27-cp27mu-manylinux1_x86_64.whl', u'bottle-0.12.7-py2-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py2-none-any.whl', u'retrying-1.3.3-py3-none-any.whl', u'cloudify_utilities_plugin-1.23.5-py3-none-any.whl', u'pycdlib-1.10.0-py2.py3-none-any.whl', u'pysnmp-4.4.5-py2.py3-none-any.whl', u'GitPython-3.1.3-py3-none-any.whl', u'MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl', u'pika-0.11.2-py2.py3-none-any.whl', u'bottle-0.12.7-py3-none-any.whl', u'chardet-3.0.4-py2.py3-none-any.whl', u'pyasn1-0.4.8-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py2-none-any.whl', u'proxy_tools-0.1.0-py2-none-any.whl', u'proxy_tools-0.1.0-py3-none-any.whl', u'GitPython-2.1.15-py2.py3-none-any.whl', u'bcrypt-3.1.7-cp27-cp27mu-manylinux1_x86_64.whl', u'MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl', u'ruamel.yaml.clib-0.2.0-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.4-py3-none-any.whl', u'PyNaCl-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl', u'ruamel.yaml-0.16.10-py2.py3-none-any.whl', u'requests-2.24.0-py2.py3-none-any.whl', u'gitdb-0.6.4-cp27-cp27mu-linux_x86_64.whl', u'smmap2-3.0.1-py2-none-any.whl', u'networkx-1.9.1-py2.py3-none-any.whl', u'cloudify_utilities_plugins_sdk-0.0.25-py3-none-any.whl', u'cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl', u'enum34-1.1.10-py2-none-any.whl', u'ruamel.ordereddict-0.4.14-cp27-cp27mu-manylinux1_x86_64.whl', u'networkx-2.2-py2.py3-none-any.whl', u'pycryptodomex-3.9.8-cp36-cp36m-manylinux1_x86_64.whl', u'cffi-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl', u'Jinja2-2.10-py2.py3-none-any.whl', u'cloudify_common-5.0.5.1-py2-none-any.whl', u'PyYAML-3.10-cp36-cp36m-linux_x86_64.whl', u'cryptography-2.9.2-cp27-cp27mu-manylinux2010_x86_64.whl', u'certifi-2020.6.20-py2.py3-none-any.whl', u'urllib3-1.25.9-py2.py3-none-any.whl', u'requests_toolbelt-0.8.0-py2.py3-none-any.whl', u'ruamel.yaml.clib-0.2.0-cp36-cp36m-manylinux1_x86_64.whl', u'gitdb2-2.0.6-py2.py3-none-any.whl'], u'excluded_wheels': None}]

There is a high risk of users encountering this issue regularly. I have experienced it two to three times in the last few days.

Steps to Reproduce

Environment:
OS (CLI), HA cluster, cloud provider
------------------------------------

Steps to reproduce:
------------------
1.
2.
3.

Expected result:
---------------

Actual result:
-------------

Why Propose Close?

None

Activity

Show:
Alex Molev
July 22, 2020, 12:24 PM

I guess it’s related to lazy plugin install, can we wrap it with a pretty message

Assignee

Łukasz Maksymczuk

Reporter

Trammell -

Severity

Critical

Target Version

5.1

Premium Only

no

Found In Version

5.1

QA Owner

None

Bug Type

unknown

Customer Encountered

No

Customer Name

None

Release Notes

no

Priority

High

Epic Link

Sprint

None

Priority

Critical
Configure