Event Console: higher notifications throughput (by asynchronous handling)

3 votes

Problem Statement:
In Event Console when using the "send monitoring notification" action the overall Event Console (mkeventd) throughput decreases dramatically (to about 1,4 eps), since all events are processed sequentially and the "cmk --notify" action is being processed synchronously.

In many cases, it is not necessary to wait for the result of a notification-handling action before processing the next event, especially when that next event may also trigger notifications. Allowing a certain level of concurrency in EC actions similar to how the notification subsystem mknotifyd already supports parallel processing would significantly improve throughput. Logwatch (log file monitoring) forwarding or SNMP monitoring would be typical use cases to benefit from this.

Suggestion:
By executing actions in the background asynchronously the Event Console could handle events much more efficiently (light weight). This would make it possible to reach substantially higher processing rates, on the order of 50–200 events per second.

This concept is comparable to how the EC already supports asynchronous custom actions such as “Execute shell script.” However, these custom actions currently provide less context and require more manual work. Builtin support for asynchronous EC actions would therefore offer clearer structure, better context management, and much higher throughput figures than only 1,4 events per second.

Under consideration Log monitoring / Event Console Suggested by: Boris Bockstaele Upvoted: 11 May Comments: 0

Comments: 0