[icinga-users] poor performance on icinga reload (ido2db)

Marco Hoyer m.hoyer at ecktown.net
Thu Mar 21 15:25:51 CET 2013

Hello folks,

I'm currently planning a setup of Icinga checking some thousands of services (maybe 8-10k) on round about 1500 hosts.
There is no big problem at all regarding the amount of checks but on icinga reload there is a wait time of actually 25s where there is nothing going on (with 300 hosts and 3000 services). 
Ido2db is initializing the status database with the new config during this time. The problem is, that I would like to reload icinga everytime a config-change is made. Therefor a wait time of maybe 5s would be acceptable.
Is there anyone having a similar setup and maybe an idea, how to fix this? 
The mysql database storing the icinga status db is actually capable of ap. 1200 queries/sec which seems to be very well.
We think about using SSD storage for it to increase iops but there should be a better solution than throwing hardware on it.

My ideas:
- I would like to reduce the amount of data to be loaded by reducing the data_processing_opts value in idomod.cfg. May there be a good value for icinga-web usage?
- Another idea was to use set "clean_realtime_tables_on_core_startup=0" and "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no big effort. Are these parameters working properly?

Is there anything left I can tune to get a better performance?

To prevent some questions: 
We generate icinga config-snipplets on the monitored system and push it to the icinga instances. In case of an update on maybe 50 webserver nodes there may be some config we need very fast to be active in icinga.
To prevent an ongoing reload of icinga, we do it only once per update transaction. It means that an update on all webservers only causes one reload. But doing CLD on many of our systems means that there might be many updates 
and therefor many reloads of icinga. 

Used versions (compiled from source for RHEL6 and packaged as RPM using the supplied spec file):
Icinga 1.8.4
Icinga-Web 1.8.2

kind regards and thank you for your help
Marco Hoyer

More information about the icinga-users mailing list