Events listing with UI is sub-optimal and might need improvement
A page <cloudify-ui>/console/page/logs fetches all the records from events and logs tables, combining them with four or five others. https://github.com/cloudify-cosmo/cloudify-manager/blob/70b2aa4e6826a6b7ecf0e62d4b36861ed21efc98/rest-service/manager_rest/rest/resources_v1/events.py#L210
We ( and ) tried optimizing it but faced with:
issues with pagination sorting/limiting/offsetting: there is no easy way to limit the records being fetched at the very beginning of building a query since it is union-ed data that is later sort-ed/limit-ed/ofset-ed,
issues with pagination: every time the UI request is sent, all the combined events and logs records, joined with four or five another tables are being count-ed
All of the above make some ideas described in https://gist.github.com/mateumann/c417048011636d627b05e3bbf5d27b80 obsolete. We just cannot quickly count over 15 millions of records over five to seven SQL tables.
So two ideas popped up:
get rid of this view from UI completely (who needs to browse all the logs, from all the deployments, page by page, being able at the same time to jump exactly to page fifteen hundreds?)
add some kind of counter-cache over those large tables that would allow us to get rid of count(*) on every GET /events request
change pagination on the logs page for one based no on limit/offset but limit/greater-than, i.e. select 50 elements with timestamp greater than this one, sorted by timestamp
Look at how pagination works at Google’s search service, or a pi-hole.