[icinga-users] Antwort: Re: poor performance on icinga reload (ido2db)

Jacques.Mineur at drv-bund.de Jacques.Mineur at drv-bund.de
Fri Mar 22 07:38:01 CET 2013


Hi

>> - Another idea was to use set "clean_realtime_tables_on_core_startup=0"
and "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no
big effort. Are these parameters working properly?

>they are tagged experimental as for the reason being exactly that. i had
>no further reports expect one bug to those options, so unless people
>report it working properly, i have no intention to remove the
>experimental tag at all.

we are using this option every day.
we don't do in normal case any cleanup over the day.
One time in the night, we restart icinga with the options clean*=1
(important to remove the services or hosts that we have deleted)

We can say that this options are stabil. We don't had any problems in the
past.

Jacques Mineur (ossmon)



Von:	Michael Friedrich <michael.friedrich at gmail.com>
An:	icinga-users at lists.sourceforge.net
Datum:	22.03.2013 07:28
Betreff:	Re: [icinga-users] poor performance on icinga reload (ido2db)



On 21.03.2013 15:25, Marco Hoyer wrote:
> Hello folks,
>
> I'm currently planning a setup of Icinga checking some thousands of
services (maybe 8-10k) on round about 1500 hosts.
> There is no big problem at all regarding the amount of checks but on
icinga reload there is a wait time of actually 25s where there is nothing
going on (with 300 hosts and 3000 services).


25s isn't that much. my dev box with a variety of config specialities
for testing all possible configurations takes ~80s with 4k services on a
dualcore vm. no tuning applied to mysql 5.1.66 on debian squeeze.

for the problem itsself - there have been so many discussions in the
past on the mailinglists / forums / dev tracker, that i won't repeat
them here.

for the configuration reload in general, this is required and the
problem is well known.

more information to be found here: https://dev.icinga.org/issues/1934


> Ido2db is initializing the status database with the new config during
this time. The problem is, that I would like to reload icinga everytime a
config-change is made. Therefor a wait time of maybe 5s would be
acceptable.
> Is there anyone having a similar setup and maybe an idea, how to fix
this?

that's not fixable within the current architecture. the core does not
send diffs, as it just does not do any on a reload. so the application
(ido2db) is left alone doing update/insert logic. bad design, but
requires a full rewrite which won't happen within 1.x

again, read #1934 for detailed thoughts and information.

> The mysql database storing the icinga status db is actually capable of
ap. 1200 queries/sec which seems to be very well.
> We think about using SSD storage for it to increase iops but there should
be a better solution than throwing hardware on it.

ssd might be fast, but also broken fast enough with many write ops over
time. you've been warned.


>
> My ideas:
> - I would like to reduce the amount of data to be loaded by reducing the
data_processing_opts value in idomod.cfg. May there be a good value for
icinga-web usage?

the one which is set by default. read the configuration comments, as
well as follow the wiki for some more proposals on tuning that.

> - Another idea was to use set "clean_realtime_tables_on_core_startup=0"
and "clean_config_tables_on_core_startup=0" in ido2db.cfg but there was no
big effort. Are these parameters working properly?

they are tagged experimental as for the reason being exactly that. i had
no further reports expect one bug to those options, so unless people
report it working properly, i have no intention to remove the
experimental tag at all.

>
> Is there anything left I can tune to get a better performance?

show some love to your database, and invest some time in reading tuning
guides. the icinga wiki holds a bunch of such, which i have collected
over the past years in order to resolve the general interest "where is
the tuning guide?".

well, read and learn here:
https://wiki.icinga.org/display/howtos/Performance+Tuning

>
> To prevent some questions:
> We generate icinga config-snipplets on the monitored system and push it
to the icinga instances. In case of an update on maybe 50 webserver nodes
there may be some config we need very fast to be active in icinga.
> To prevent an ongoing reload of icinga, we do it only once per update
transaction. It means that an update on all webservers only causes one
reload. But doing CLD on many of our systems means that there might be many
updates
> and therefor many reloads of icinga.
>
> Used versions (compiled from source for RHEL6 and packaged as RPM using
the supplied spec file):
> Icinga 1.8.4
> Icinga-Web 1.8.2

1.9 will provide some - again tagged experimental - options to tune the
reload in terms of making it faster.

https://dev.icinga.org/issues/3527
https://dev.icinga.org/issues/3533

still, the real bottleneck is the database and even if the socket_queue
will help resolve minimize reloading time, it will

a) cut ido2db more cpu cycles and memory
b) create sort of a proxy which means buffering the data, while waiting
for the db to finish
c) the data arrival in the database will still be slow, so web guis
might still show old data during that update process

since it's another component between icinga and the database, it might
contain unwanted behaviour, so it will be experimental and turned off by
default.

the transaction patch encapsulates a config object and its relations
(contacts, dependencies, etc) into a single transaction. this is not
perfect playing but still shows some huge performance increasements in
special environments. still, in my tests it did not help that much, so
it's experimential either way - and turned off by default.

though, 1.9 isn't release yet, but you may test the beta version, which
is due 1 week before release (planned 25.4.)

kind regards,
Michael


PS: Icinga2 will add ido support via compat module too, in order to stay
compatible with Icinga Web. That's a todo for m3.


--
DI (FH) Michael Friedrich

mail:     michael.friedrich at gmail.com
twitter:  https://twitter.com/dnsmichi
jabber:   dnsmichi at jabber.ccc.de
irc:      irc.freenode.net/icinga dnsmichi

icinga open source monitoring
position: lead core developer
url:      https://www.icinga.org

------------------------------------------------------------------------------

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
icinga-users mailing list
icinga-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/icinga-users






More information about the icinga-users mailing list