[icinga-users] Scaling Icinga and Passive checks

Matuskiewicz, Philip Philip.Matuskiewicz at nyct.com
Tue Mar 12 14:45:42 CET 2013


Hi,

I am experiencing difficulties in scaling Icinga to 6,000 hosts and roughly 120,000 services.  I'm curious if anyone here has any advice on how I could achieve this kind of scale, specifically on the classic user interface.  Regardless of Icinga doing checks or not (I can disable active and passive checks), the classic UI remains extremely slow and nearly useless (15 seconds per load, regardless of how big the host machine is).  This is running on an Amazon AWS instance, and the UI is slow regardless of how big I make the instance (I've tried using a 30GB Ram instance with high disk io, memory io, and cpu usage to give you perspective).

A little background information about configuration:

The environment uses SNMP exclusively, I have written some surrounding applications to accept SNMP traps and inform messages at this scale and translate them to the format that Icinga's external command file requires.  To get these messages to the command file, I use ZeroMQ to ensure that only one command can be appended at a time to the file.  Freshness checking is enabled on everything for a period of 60 minutes, and the active check is just a bash script that returns "its down" with the service / host critical / down code.  I have also moved the status.dat, caches, and checkresults to a folder located on ram using suggestions from elsewhere.  Outside of a vanilla Icinga installation, the modifications above, and the addition of MKLiveStatus, I have done nothing else from the defaults.  From my initial testing, I believe that Icinga is mostly keeping up with the flow of commands in this configuration (roughly an average 25 commands/second, peaking around 100 commands/second).

In my testing, I have looked at Icinga-web to pull the interface away from the core server, but the MySQL connector (idoutils) seems to be extremely buggy upon start-up, sometimes taking an hour or more to start getting the live flow of data from the core, it also places a fair amount of strain on any MySQL server that I throw at it.

Could anyone give me advice on how to proceed or optimize Icinga so that the interface works a little faster?

Thanks,
Phil

Philip Matuskiewicz
Systems Developer - MTA Bus Time (http://bustime.mta.info)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.icinga.org/pipermail/icinga-users/attachments/20130312/2205a04d/attachment.html>


More information about the icinga-users mailing list