Events listing with UI is sub-optimal and might need improvement

Description

A page <cloudify-ui>/console/page/logs fetches all the records from events and logs tables, combining them with four or five others. https://github.com/cloudify-cosmo/cloudify-manager/blob/70b2aa4e6826a6b7ecf0e62d4b36861ed21efc98/rest-service/manager_rest/rest/resources_v1/events.py#L210

We ( and ) tried optimizing it but faced with:

  • issues with pagination sorting/limiting/offsetting: there is no easy way to limit the records being fetched at the very beginning of building a query since it is union-ed data that is later sort-ed/limit-ed/ofset-ed,

  • issues with pagination: every time the UI request is sent, all the combined events and logs records, joined with four or five another tables are being count-ed

All of the above make some ideas described in https://gist.github.com/mateumann/c417048011636d627b05e3bbf5d27b80 obsolete. We just cannot quickly count over 15 millions of records over five to seven SQL tables.

So two ideas popped up:

  • get rid of this view from UI completely (who needs to browse all the logs, from all the deployments, page by page, being able at the same time to jump exactly to page fifteen hundreds?)

  • add some kind of counter-cache over those large tables that would allow us to get rid of count(*) on every GET /events request

  • change pagination on the logs page for one based no on limit/offset but limit/greater-than, i.e. select 50 elements with timestamp greater than this one, sorted by timestamp

Assignee

Unassigned

Reporter

Mateusz Neumann

Labels

Target Version

future

QA Owner

None

Premium Only

no

Documentation Required

None

Why Blocked?

None

Release Notes

yes

Priority

Medium

Epic Link

Priority

Medium
Configure