[icinga-users] poor performance on icinga reload (ido2db)
michael.friedrich at gmail.com
Thu Mar 21 21:48:51 CET 2013
On 21.03.2013 16:13, Michael Friedrich wrote:
> On 21.03.2013 15:25, Marco Hoyer wrote:
>> Hello folks,
>> I'm currently planning a setup of Icinga checking some thousands of
>> services (maybe 8-10k) on round about 1500 hosts.
>> There is no big problem at all regarding the amount of checks but on
>> icinga reload there is a wait time of actually 25s where there is
>> nothing going on (with 300 hosts and 3000 services).
> 25s isn't that much. my dev box with a variety of config specialities
> for testing all possible configurations takes ~80s with 4k services on a
> dualcore vm. no tuning applied to mysql 5.1.66 on debian squeeze.
> for the problem itsself - there have been so many discussions in the
> past on the mailinglists / forums / dev tracker, that i won't repeat
> them here.
> for the configuration reload in general, this is required and the
> problem is well known.
> more information to be found here: https://dev.icinga.org/issues/1934
>> Ido2db is initializing the status database with the new config during
>> this time. The problem is, that I would like to reload icinga
>> everytime a config-change is made. Therefor a wait time of maybe 5s
>> would be acceptable.
>> Is there anyone having a similar setup and maybe an idea, how to fix
> that's not fixable within the current architecture. the core does not
> send diffs, as it just does not do any on a reload. so the application
> (ido2db) is left alone doing update/insert logic. bad design, but
> requires a full rewrite which won't happen within 1.x
> again, read #1934 for detailed thoughts and information.
>> The mysql database storing the icinga status db is actually capable of
>> ap. 1200 queries/sec which seems to be very well.
>> We think about using SSD storage for it to increase iops but there
>> should be a better solution than throwing hardware on it.
> ssd might be fast, but also broken fast enough with many write ops over
> time. you've been warned.
>> My ideas:
>> - I would like to reduce the amount of data to be loaded by reducing
>> the data_processing_opts value in idomod.cfg. May there be a good
>> value for icinga-web usage?
> the one which is set by default. read the configuration comments, as
> well as follow the wiki for some more proposals on tuning that.
seems i wrote a wiki entry for that as well. someone should monitor my
wiki article count...
>> - Another idea was to use set
>> "clean_realtime_tables_on_core_startup=0" and
>> "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no
>> big effort. Are these parameters working properly?
> they are tagged experimental as for the reason being exactly that. i had
> no further reports expect one bug to those options, so unless people
> report it working properly, i have no intention to remove the
> experimental tag at all.
>> Is there anything left I can tune to get a better performance?
> show some love to your database, and invest some time in reading tuning
> guides. the icinga wiki holds a bunch of such, which i have collected
> over the past years in order to resolve the general interest "where is
> the tuning guide?".
> well, read and learn here:
>> To prevent some questions:
>> We generate icinga config-snipplets on the monitored system and push
>> it to the icinga instances. In case of an update on maybe 50 webserver
>> nodes there may be some config we need very fast to be active in icinga.
>> To prevent an ongoing reload of icinga, we do it only once per update
>> transaction. It means that an update on all webservers only causes one
>> reload. But doing CLD on many of our systems means that there might be
>> many updates
>> and therefor many reloads of icinga.
>> Used versions (compiled from source for RHEL6 and packaged as RPM
>> using the supplied spec file):
>> Icinga 1.8.4
>> Icinga-Web 1.8.2
> 1.9 will provide some - again tagged experimental - options to tune the
> reload in terms of making it faster.
> still, the real bottleneck is the database and even if the socket_queue
> will help resolve minimize reloading time, it will
> a) cut ido2db more cpu cycles and memory
> b) create sort of a proxy which means buffering the data, while waiting
> for the db to finish
> c) the data arrival in the database will still be slow, so web guis
> might still show old data during that update process
> since it's another component between icinga and the database, it might
> contain unwanted behaviour, so it will be experimental and turned off by
> the transaction patch encapsulates a config object and its relations
> (contacts, dependencies, etc) into a single transaction. this is not
> perfect playing but still shows some huge performance increasements in
> special environments. still, in my tests it did not help that much, so
> it's experimential either way - and turned off by default.
> though, 1.9 isn't release yet, but you may test the beta version, which
> is due 1 week before release (planned 25.4.)
> kind regards,
> PS: Icinga2 will add ido support via compat module too, in order to stay
> compatible with Icinga Web. That's a todo for m3.
DI (FH) Michael Friedrich
mail: michael.friedrich at gmail.com
jabber: dnsmichi at jabber.ccc.de
irc: irc.freenode.net/icinga dnsmichi
icinga open source monitoring
position: lead core developer
More information about the icinga-users