AMQP client not closing connections correctly

Description

During or after deployment creation, errors are logged in rabbit regarding connections being dropped without being closed:

This suggests that one of our AMQP clients is not correctly closing connections.

Currently this does not have a major impact, but it does cause log noise and it is unclear without investigation whether this would begin to cause issues at higher manager load (e.g. due to ports which are no longer in use waiting to be closed).

Activity

Show:
Michael Sverdlik
December 11, 2015, 2:14 PM

This happens because amqp-influx is running before influxdb. It's possible to suppress these errors by creating the right rules in systemd's unit file to make sure amqp-influx is up and running only after influxdb and rabbitmq are running. However it still takes few moments for the actual port to be open so I'm not sure we will entirely remove these errors from logs.

geokala
December 11, 2015, 5:15 PM

Uncertain on that- I gave it a quick test wrapping that code with try/except and restarting amqpinflux on a test manager yesterday and it still showed the same errors coming up in rabbit's logs, but nothing on my hacky debug output. Might've been messed up debug output, of course, but could also be somewhere else causing this.

Isaac Shabtay
December 14, 2015, 7:08 PM

This may have something to do that the AMQPClient instance used on a per-thread basis never actually closes its connection:

https://github.com/cloudify-cosmo/cloudify-plugins-common/blob/master/cloudify/amqp_client.py#L79

The close() function is never called. We found this in 3.2.1, on a client installation. This bug also exists in 3.3 and is currently being investigated.

Michael Sverdlik
December 16, 2015, 3:56 PM

Duplicate

Assignee

Ran Ziv

Reporter

geokala

Labels

Bug Type

None

Target Version

None

Severity

None

Sprint

None

Fix versions

Affects versions

Configure