[icinga-users] poor performance on icinga reload (ido2db)

Michael Friedrich michael.friedrich at gmail.com
Thu Mar 21 16:13:32 CET 2013


On 21.03.2013 15:25, Marco Hoyer wrote:
> Hello folks,
>
> I'm currently planning a setup of Icinga checking some thousands of services (maybe 8-10k) on round about 1500 hosts.
> There is no big problem at all regarding the amount of checks but on icinga reload there is a wait time of actually 25s where there is nothing going on (with 300 hosts and 3000 services).


25s isn't that much. my dev box with a variety of config specialities 
for testing all possible configurations takes ~80s with 4k services on a 
dualcore vm. no tuning applied to mysql 5.1.66 on debian squeeze.

for the problem itsself - there have been so many discussions in the 
past on the mailinglists / forums / dev tracker, that i won't repeat 
them here.

for the configuration reload in general, this is required and the 
problem is well known.

more information to be found here: https://dev.icinga.org/issues/1934


> Ido2db is initializing the status database with the new config during this time. The problem is, that I would like to reload icinga everytime a config-change is made. Therefor a wait time of maybe 5s would be acceptable.
> Is there anyone having a similar setup and maybe an idea, how to fix this?

that's not fixable within the current architecture. the core does not 
send diffs, as it just does not do any on a reload. so the application 
(ido2db) is left alone doing update/insert logic. bad design, but 
requires a full rewrite which won't happen within 1.x

again, read #1934 for detailed thoughts and information.

> The mysql database storing the icinga status db is actually capable of ap. 1200 queries/sec which seems to be very well.
> We think about using SSD storage for it to increase iops but there should be a better solution than throwing hardware on it.

ssd might be fast, but also broken fast enough with many write ops over 
time. you've been warned.


>
> My ideas:
> - I would like to reduce the amount of data to be loaded by reducing the data_processing_opts value in idomod.cfg. May there be a good value for icinga-web usage?

the one which is set by default. read the configuration comments, as 
well as follow the wiki for some more proposals on tuning that.

> - Another idea was to use set "clean_realtime_tables_on_core_startup=0" and "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no big effort. Are these parameters working properly?

they are tagged experimental as for the reason being exactly that. i had 
no further reports expect one bug to those options, so unless people 
report it working properly, i have no intention to remove the 
experimental tag at all.

>
> Is there anything left I can tune to get a better performance?

show some love to your database, and invest some time in reading tuning 
guides. the icinga wiki holds a bunch of such, which i have collected 
over the past years in order to resolve the general interest "where is 
the tuning guide?".

well, read and learn here: 
https://wiki.icinga.org/display/howtos/Performance+Tuning

>
> To prevent some questions:
> We generate icinga config-snipplets on the monitored system and push it to the icinga instances. In case of an update on maybe 50 webserver nodes there may be some config we need very fast to be active in icinga.
> To prevent an ongoing reload of icinga, we do it only once per update transaction. It means that an update on all webservers only causes one reload. But doing CLD on many of our systems means that there might be many updates
> and therefor many reloads of icinga.
>
> Used versions (compiled from source for RHEL6 and packaged as RPM using the supplied spec file):
> Icinga 1.8.4
> Icinga-Web 1.8.2

1.9 will provide some - again tagged experimental - options to tune the 
reload in terms of making it faster.

https://dev.icinga.org/issues/3527
https://dev.icinga.org/issues/3533

still, the real bottleneck is the database and even if the socket_queue 
will help resolve minimize reloading time, it will

a) cut ido2db more cpu cycles and memory
b) create sort of a proxy which means buffering the data, while waiting 
for the db to finish
c) the data arrival in the database will still be slow, so web guis 
might still show old data during that update process

since it's another component between icinga and the database, it might 
contain unwanted behaviour, so it will be experimental and turned off by 
default.

the transaction patch encapsulates a config object and its relations 
(contacts, dependencies, etc) into a single transaction. this is not 
perfect playing but still shows some huge performance increasements in 
special environments. still, in my tests it did not help that much, so 
it's experimential either way - and turned off by default.

though, 1.9 isn't release yet, but you may test the beta version, which 
is due 1 week before release (planned 25.4.)

kind regards,
Michael


PS: Icinga2 will add ido support via compat module too, in order to stay 
compatible with Icinga Web. That's a todo for m3.


-- 
DI (FH) Michael Friedrich

mail:     michael.friedrich at gmail.com
twitter:  https://twitter.com/dnsmichi
jabber:   dnsmichi at jabber.ccc.de
irc:      irc.freenode.net/icinga dnsmichi

icinga open source monitoring
position: lead core developer
url:      https://www.icinga.org




More information about the icinga-users mailing list