[icinga-users] poor performance on icinga reload (ido2db)

Marco Hoyer m.hoyer at ecktown.net
Thu Mar 21 18:33:02 CET 2013

Am 21.03.2013 um 16:13 schrieb Michael Friedrich:

> On 21.03.2013 15:25, Marco Hoyer wrote:
>> Hello folks,
>> I'm currently planning a setup of Icinga checking some thousands of services (maybe 8-10k) on round about 1500 hosts.
>> There is no big problem at all regarding the amount of checks but on icinga reload there is a wait time of actually 25s where there is nothing going on (with 300 hosts and 3000 services).
> 25s isn't that much. my dev box with a variety of config specialities 
> for testing all possible configurations takes ~80s with 4k services on a 
> dualcore vm. no tuning applied to mysql 5.1.66 on debian squeeze.

If the goal is to have double the hosts and services per host, a reload may take 50s. That's a big problem if you have to reload quite often.
Our infrastructure is in a process of permanent change. Additional load causes the creation of new machines, apps get updates some times a day.
All this may cause icinga config changes.

> for the problem itsself - there have been so many discussions in the 
> past on the mailinglists / forums / dev tracker, that i won't repeat 
> them here.

I know about some of them but things change and there might be something I haven't heard about.

> for the configuration reload in general, this is required and the 
> problem is well known.
> more information to be found here: https://dev.icinga.org/issues/1934
>> Ido2db is initializing the status database with the new config during this time. The problem is, that I would like to reload icinga everytime a config-change is made. Therefor a wait time of maybe 5s would be acceptable.
>> Is there anyone having a similar setup and maybe an idea, how to fix this?
> that's not fixable within the current architecture. the core does not 
> send diffs, as it just does not do any on a reload. so the application 
> (ido2db) is left alone doing update/insert logic. bad design, but 
> requires a full rewrite which won't happen within 1.x
> again, read #1934 for detailed thoughts and information.
>> The mysql database storing the icinga status db is actually capable of ap. 1200 queries/sec which seems to be very well.
>> We think about using SSD storage for it to increase iops but there should be a better solution than throwing hardware on it.
> ssd might be fast, but also broken fast enough with many write ops over 
> time. you've been warned.
>> My ideas:
>> - I would like to reduce the amount of data to be loaded by reducing the data_processing_opts value in idomod.cfg. May there be a good value for icinga-web usage?
> the one which is set by default. read the configuration comments, as 
> well as follow the wiki for some more proposals on tuning that.

I did but am not informed enough about the side-effects on icinga-web.
There is something exported by ido2db, which seems to be unused in icinga-web like the command-definitions
but before changing this and running into a trap sometimes later, I wanted to ask somebody knowing about the requirements  of icinga-web to ido2db.

>> - Another idea was to use set "clean_realtime_tables_on_core_startup=0" and "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no big effort. Are these parameters working properly?
> they are tagged experimental as for the reason being exactly that. i had 
> no further reports expect one bug to those options, so unless people 
> report it working properly, i have no intention to remove the 
> experimental tag at all.
>> Is there anything left I can tune to get a better performance?
> show some love to your database, and invest some time in reading tuning 
> guides. the icinga wiki holds a bunch of such, which i have collected 
> over the past years in order to resolve the general interest "where is 
> the tuning guide?".
> well, read and learn here: 
> https://wiki.icinga.org/display/howtos/Performance+Tuning
>> To prevent some questions:
>> We generate icinga config-snipplets on the monitored system and push it to the icinga instances. In case of an update on maybe 50 webserver nodes there may be some config we need very fast to be active in icinga.
>> To prevent an ongoing reload of icinga, we do it only once per update transaction. It means that an update on all webservers only causes one reload. But doing CLD on many of our systems means that there might be many updates
>> and therefor many reloads of icinga.
>> Used versions (compiled from source for RHEL6 and packaged as RPM using the supplied spec file):
>> Icinga 1.8.4
>> Icinga-Web 1.8.2
> 1.9 will provide some - again tagged experimental - options to tune the 
> reload in terms of making it faster.
> https://dev.icinga.org/issues/3527
> https://dev.icinga.org/issues/3533
> still, the real bottleneck is the database and even if the socket_queue 
> will help resolve minimize reloading time, it will
> a) cut ido2db more cpu cycles and memory
> b) create sort of a proxy which means buffering the data, while waiting 
> for the db to finish
> c) the data arrival in the database will still be slow, so web guis 
> might still show old data during that update process
> since it's another component between icinga and the database, it might 
> contain unwanted behaviour, so it will be experimental and turned off by 
> default.
> the transaction patch encapsulates a config object and its relations 
> (contacts, dependencies, etc) into a single transaction. this is not 
> perfect playing but still shows some huge performance increasements in 
> special environments. still, in my tests it did not help that much, so 
> it's experimential either way - and turned off by default.
> though, 1.9 isn't release yet, but you may test the beta version, which 
> is due 1 week before release (planned 25.4.)
> kind regards,
> Michael
> PS: Icinga2 will add ido support via compat module too, in order to stay 
> compatible with Icinga Web. That's a todo for m3.

Thank you for your help and some useful hints.
I will have a look at Icinga 1.9 to see if there will be some improvement.

kind regards

> -- 
> DI (FH) Michael Friedrich
> mail:     michael.friedrich at gmail.com
> twitter:  https://twitter.com/dnsmichi
> jabber:   dnsmichi at jabber.ccc.de
> irc:      irc.freenode.net/icinga dnsmichi
> icinga open source monitoring
> position: lead core developer
> url:      https://www.icinga.org
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> icinga-users mailing list
> icinga-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/icinga-users

More information about the icinga-users mailing list