[icinga-checkins] icinga.org: icinga2/feature/configconvert-2743: Config Conversion: add conversion script

git at icinga.org git at icinga.org
Mon Jun 24 15:19:45 CEST 2013


Module: icinga2
Branch: feature/configconvert-2743
Commit: d050ee09f812bf1a912d35927cda1bb33ed3165b
URL:    https://git.icinga.org/?p=icinga2.git;a=commit;h=d050ee09f812bf1a912d35927cda1bb33ed3165b

Author: Michael Friedrich <michael.friedrich at netways.de>
Date:   Tue Mar 12 17:40:53 2013 +0100

Config Conversion: add conversion script

Squashed Rebase, commit msg below.

convertv1->v2 initial import

refs #2743

itl: fix inclusion of notification.conf

Config Conversion: command macros, templates

refs #2743

WIP: try resolving the host->svc relation from templates and tricks

not finished yet, there's heavy recursion required

WIP configconversion: rewrite recursive lookup of linked host_name/service_description

this is a mess. configs may just use "name" instead of "host_name" on
type "host", same goes for service and service_descriptions. well, and
if it's a template, "name" is populated too.

the algorithm is not clean enough, and does not scale at all. there are
bugs, and the hostgroup linking is not yet taken into account too.

basics host -> service dumping of v2 config is implemented.

refs #2743

ConfigConvert: split into modules, add objects.cache read, dump cmd line

basically it works with

- dumping a host with all services, using templates
- check_command and all arguments/user macros translation
- reading the objects.cache file location from icinga.cfg
=> this is reserved only for getting dedicated relations
=> we do not want to duplicate the work of xodtemplate.c in perl here
- all important functions have been split up into their respective
  modules in order to support better re-usage (i.e. lconfexport)

still missing important bits:

- detect service->group<-host links and build templates
- detect other ugly template methods
- dump templates
- the ugly deprecated normal|retry_check_interval mapping
- timeperiod mapping
- contact => user mapping
- notifications rework (dedicated object)
- dependencies, escalations (to be documented/implemented)
- the infamous rest

ConfigConvert: fix hash keys for convert/dump in perl 5.14

ConfigConvert: improved template handling, host/service dump 2x

refactored code again, removed nested loop bottlenecks. still, the
resolval of host_name/service_description requires more love.

templates are now correctly detected, linked, and printed in 2x syntax.
service templates are _not_ linked against host definitions in 2x as
this is not possible.

commands for hosts do not exist in 2x, therefore the command_name of an
1x host will be taken and guessed against its related services. if
there's a match, the service will be linked as 2x 'hostcheck'.

*_interval still needs proper handling, as the default interval is 1m,
but 2x requires the identifier.

ConfigConvert: refactor host_name/service_description link resolving

previous algorithm was buggy, as well get_host_name always looked up
host template objects, even if the template object was a service. this
is now fixed by re-using the __TYPE attribute (set on Icinga 1x config
read).

the algorithm is now like

                       |---------------------------------|
                      \/                                 |
                   svc_desc?                             |
                0 /        \ 1                           |
              use tmpl?    END -------> [svc_desc]       |
             0 /   \ 1                                   |
[undef] <- BROKEN  getobjbyattr(['use'], __TYPE, svcdesc |
                    |                                    |
                 getsvcdesc(obj)                         |
                    |                                    |
                    --------------------------------------

refs #2743

ConvertConfig: add basic support for users, timeperiods, *groups

as well as display_name as attribute to print something.

contact => user mapping is added as well, manipulating the __TYPE var
too. furthermore display_name on the host will be overwritten by the
alias, if set.

the groups don't dump much, only the display_name if set. their dump
function is implemented generic, based on the __TYPE

ConfigConvert: fix 4 little big thinkos on import/convert/dump

1) do not skip lines containing ;.* - there may actually be a comment on
attr->val line. rather clean the line safely, and let the magic happen.

2) the type counter was implemented in order to have unique object ids
by type, since 1x config does not provide us with such. problem - the
function had a local var, saving the counter for each file, overwriting
the hash attributes later on, starting again with 0. nasty bug, hard to
figure out. fixed by feeding the type_cnt into the cfg_obj hive for now.
revamped the state machine logic as well, like so

I: define TYPE { 	--> __TYPE = TYPE
II: attr val		--> {type}->{type_cnt->cnt}->{attr} = val
III: }			--> type_cnt->cnt++
IV: <empty/comment>

3) also skip service templates (__IS_TEMPLATE) sooner, and do not try to
calculate some host_name/service_description magic out of them - they
just can't go down one level to the host, but only up with 'use'.

4) host and service templates on 2x config dump must check if
__USES_TEMPLATE is actually set to 1, checking for defined is not
enough.

whatthecommit.com:

A long time ago, in a galaxy far far away...

refs #2743

ConfigConvert: add more host/service attributes, groups

*_interval are automatically mapped, while the *groups attribute it
converted into an array in order to easily stash more groups on it.

still a todo - loop over existing groups and re-link their members into
the respective objects, as 2.x only supports that location.

fix for the host->service relation - use the already processed service
2x hashref in order to do some magic with host->{'SERVICE'} = service

all 2x attributes are now encapsulated with "", to stay safe.
todo - more generic config 2x dump.

furthermore, add some more code comments for the respective sections.

refs #2743

ConfigConvert: fix service linking not unique, add hostgroup->members re-link to host->hostgroups

while working on the missing servicegroups and their facepalm members
syntax (host1,svc1,host2,svc2 - awesome to parse), i figured that the
whole detection of services by their attribute 'service_description' is
not uniquely implemented. now the getter functions are duplicated, and
not modular anymore, for the sake of having a service identified by its
host_name + service_description. since that lookup is recursive, we must
pass the host_name from the very beginning - otherwise we could
accidently overwrite that with a template entry.

the hostgroup re-linking works, also thanks to the fact that the inner
object structure is just an array in the hashref, which allows smart
push.

contact/user groups are tricky again due to the different naming.

further todo: sort all macros and hostservice links. right now this
looks like chaos.

refs #2743

ConfigConvert: basic servicegroup parsing method

ConfigConvert: refactor multiple host_name's on service object detection

this is ugly. a service object may contain 'host_name' with
'host1,host2,host3' and so on. treating this as string won't allow us to
look into the template tree for valid host attributes, so we need to
rewrite the getter function, allowing some special treatment on
host_name arrays.

for the service object preparation this means: while in the service
loop, we need to loop again over all hosts, then dclone the hashref
object for the service, and add the unique host_name attribute again.

we might require some loop unrolling later, but for now this works as
expected.

refs #2743

ConvertConfig: disable debug output, some reformatting of dump

ConfigConvert: refactor host-svc links, add full service/usergroup relinking

due to the relinking afterwards we actually modify the service objects
already prepared for 2x. since the *group magic depends on the
host/services/users already prepared, we get a perpetuum mobile here
with the host loop and resolving service relations.

in order to safely fix the issue, there's a seperate host-service link
loop after resolving all the other needed objects, which may cost a
little bit more performance, but is safe and standalone, getting all
previous modifications, migrations and also hacks.

next to the refactored flow, the issue with resolving the correct
servicegroups and stashing them into an array is solved, as well as
converting the existing contactgroups to usergroups, as well as
re-linking existing contactgroup->members to their new home,
user->usergroups.

note: when parsing the servicegroup members into an array, it must not
be sorted afterwards, because we will shift 2 objects (host, service) in
order to identify the correct service object for adding the servicegroup
item.

some code reorganisation too, plus a minor comma fix for the host
address macro.

the used 1x configuration will be addressed in 2743.cfg on
icinga-core.git, with all future additions.

refs #2743

ConfigConvert: *interval, max_check_attempts dumped as numbers

see
https://git.icinga.org/?p=icinga2.git;a=blob_plain;f=lib/icinga/icinga-type.conf;hb=HEAD

refs #2743

ConfigConvert: fix undocumented contact templates conversion

refs #2743

s/DumpIcinga2Cfg/ExportIcinga2Cfg/g

just for Import/Export clarification.

configconvert: use special prefix for internal attributes

__I2CONVERT_ should do the trick, not to interfere with existing custom
variables.

configconvert: fix leftover from rename

configconvert: properly escape strings with double quotes

* command_line and command_args
* notification commands
* action_url, notes_url
* notes
* icon_image, icon_image_alt

fixes #3931

configconvert: fix multiple templates usage, template object lookup error

multiple host/service templates seperated by commas will cause some
irritation, therefore this is treated as array internally, like group
members. for the config export and "inherits", this is easy to fix. for
the actual host_name lookup for service objects, this unveiled another
bug when looping through all available templates - they are unique by
'name' and not by hostname/servicedesc, and require their very own
getter - obj_get_tmpl_obj_by_tmpl_name() - which makes sure to return
the object which is then looked into recursively if there's a host_name
provided.

possible bug for now - multiple service templates used and the host_name is
defined in one of them. but that's not the usual case, and won't be
supported for now.

refs #2743
fixes #3932

configconvert: dump modular timperiod attr and values

we need to ignore all other attributes which are to be cleaned sooner,
just for safety.

minor flaw: the keys are not sorted by weekdays now, but dumped like the
hash provides them.

fixes #4013

configconvert: support 'name' for groups' if *group_name is not set

this is a fallback for broken configs only, and since i got some, added
a small fix.

refs #2743

configconvert: add service->hostgroup<-hostmembers conversion to template support

WIP: there's multiple service->hostgroup relations - duplicated hosthg
templates. must be fixed.

refs #2743
refs #4016

configconvert: refactor service-hg-hosts for one hosthg template and multiple services

we need to first loop over all services, getting the key ids, collecting
them for each hostgroup_name (this is the unique identifier, because
this is our hosthg template later on).

then we loop over the collected unique hostgroups and build the 2.x
template logic. the difference here is now, that the code is refactored,
and loops over the service keys (getting their objects) only once,
pushing their templates plus a relation to a single (1) hosthg-template.

there's still some space for enhancements - the service template names
as well as hosthg template names as well as service object names are
generated with some hardcoded strings/numbers.

refs #2743
refs #4016

configconversion: remove debug output

configconvert: only dump macros if there are key-val pairs

configconvert: code cleanup

configconversion: explicitely check __I2CONVERT_USES_TEMPLATE == 1 when dumping service templates

refs #4020

configconvert: refactor how services without host_name are handled

this is mainly the cause when using hostgroup relations, or other object
tricks. we still need the single loop run to convert all 2.x object
attributes in one run, and deal with the missing attributes later on.

configconvert: WIP notification conversion

- refactored notification_commands parsing and storage
- do not add duplicated notification_command names to the 2x
  notification object hive
- create notifications based on the notification command linked from
  users
- move host->service link magic to the end again

refs #4008

configconvert: WIP notifications, how service/hosts relate to users and notifications

there may be more than one notification for the user, and we already had
the type in place. currently, that type is not taken into account (bug
to fix).

the rest with service -> user -> notification linking works in the first
implementation.

refs #4008

configconvert: final stage, rework notification type handling, fix naming amd template output

the config dump must take care of which notification type is dumped,
otherwise there would be yet another look required. Instead we are just
passing a reference from users to hosts/services with all prepared
notification attributes, and check them lazy on dump then.

further fixe on the notification names - they are not unique by their
command (now used on host/service scope, not user/contact) so we'll add
some magic number as suffix.

fixed template name dumping due to changed conversion vars too.

refs #4008

configconvert: write config files, refactor how objects are exported

structure is currently hardcoded, but may be changed to getopts later
on. all files are overridden into conf/ (testing only right on).

refactored export code, removed everything from Convert.pm, and export
is now called from the script directly, wrapped all dump loops and added
more function layers in betweed. open/close file handles take care of
everything, next to a header on top of the files for getting to know
their origin better (our hard work, ey?).

refs #4036

configconvert: fix service description handling, it's now an inner conversion value

previously, i've just re-used 'service_description' which isn't a valid
2.x attribute either. the lookup of service_description attribute in 1.x
templates works already, but populated __I2CONVERT_SERVICEDESCRIPTION
instead. in favor of having a generated value for all our objects, we'll
use that one, also for dumping the configuration.

even further, this is also changed for the service->hostgroup<-hosts
conversion then.

this also avoids writing empty service names into host->service objects.

fixes #4019

configconvert: add small main config file for standalone tests

refs #2743

configconvert: s/usergroups/groups/ for User config dump

configconvert: add README

configconvert: add ITL templates automatically (for services, notifications)

this currently only makes sense for services and notifications.

refs #4037

configconvert: fix timeperiod ranges, add template support for ITL inherits

refs #4037
refs #4013

configconvert: update requirements, perl deps

refs #2743

configconvert: fix import parser - timeperiods require regex (thx LConfImport)

somehow, the Icinga2 import still fails with us holiday definitions. bug
that's not yet determined whose bug that is.

refs #4039
refs #4013

configconvert: add getopts and pod2usage incl help

fixes #4041

configconvert: add notification_interval|period

and refactor check|retry_interval vs normal|retry_check_interval

refs #4008

configconvert: make host->service linked service a template object, referenced by the host

this is not the best solution, as it it creates a service template for
each host-service relation. if you have many host_name entries defined
in your Icinga 1.x config for a service, this will create duplicates
then. requires some more code refactoring, therefore leaving the issue
open.

refs #3933

configconvert: drop config file prefix 'icinga2'

just makes it hard to work with.

refs #4036

configconvert: code cleanup

configconvert: export check_period for hosts/services

refs #2743

configconvert: WIP dependencies - for host based on host_name and parents

refs #4011

configconvert: s/Timeperiod/TimePeriod/g

refs #4013

configconvert: add hostdependency with hostgroup_name hosts unrolling

refs #4011

configconvert: add servicedependency support (host_name)

missing - the loop fun with hostgroups.

special config is here:
https://git.icinga.org/?p=icinga-core.git;a=blob;f=tests/etc/2743.cfg;hb=refs/heads/next#l376

refs #4011

configconvert: add service dependency support for hostgroups

well, a lot of looping required here:

- loop through all child hostgroup hosts and get their hostnames
- get the servive object based on each hostname (loop again) and
  the child service description
- loop through all master hostgroup hosts and get their hostnames
- get the master hostname based on each hostname (loop again) and
  the master service description, save that as service dependency

given that implementation, dependencies may be converted for now.

once we decide which options we support, we may add them as well.

refs #4011

configconvert: add support for * as service host_name next to comma seperated list

this is currently a proof of concept for services only, but should
target all the object tricks later on. this will be done on host_name
lookup and stored as array.

http://docs.icinga.org/latest/en/objecttricks.html

refs #4010

configconvert: refactor commaseperated list split

... to list of stripped strings into a function.

fixes #4046

configconvert: handle excluded hosts, ignore them on split to array

exclusions/exceptions cannot be treated by icinga2 the same way as in
1.x so it's better to not create unwanted objects.

fixes #4007

configconvert: replace colon in object names with underscore

it's a special delimiter and therefore forbidden.

fixes #4225

configconvert: detect additive inheritance for *groups and use += on export

i guess this can be used for host/service contact(groups) and other foo
as well, so let's see.

refs #4006

configconvert: add wildcard detection for 'members *' in hostgroup

currently adds itself to the host's hostgroups array, but maybe a global
template where all hosts inherit from, plus additive inheritance fit
better here.

refs #4010

configconvert: add a note about servicedeps wildcards todo

refs #4010

configconvert: svg-hg-hst logic may contain multiple hgnames for services

the object tricks section on the 1.x documentation contains everything
which makes the code less readable even here.

luckily the code is rather modular, so the extension for split string to
array, and looping doesn't hurt much.

refs #4010

configconvert: update README with usage and todos

refs #2743

configconvert: fix perl 5.14, skip non-existing objects

configconvert: escape display_name (may contain quotes)

refs #3931

configconvert: refactor using command objects (check/notification)

eventhandlers are missing.

fixes #4305

configconvert: add volatile attribute

fixes #4321

configconvert: eventhandler as EventCommand objects

fixes #4327

update TODOs

configconvert: remove obsolete itl notification template

and check if defined when used.

refs #2743

configconvert: write compat log too in icinga2 conf

Revert "itl: fix inclusion of notification.conf"

This reverts commit 02eb54dc1a818fdcafa6c85817e9f6341cf2831a.

configconvert: fix thinko with check_command|event_command in service

refs #2743
refs #4327
refs #4305

---

 contrib/configconvert/Icinga2.pm                  |   35 +
 contrib/configconvert/Icinga2/Convert.pm          | 1901 +++++++++++++++++++++
 contrib/configconvert/Icinga2/ExportIcinga2Cfg.pm |  923 ++++++++++
 contrib/configconvert/Icinga2/ImportIcinga1Cfg.pm |  296 ++++
 contrib/configconvert/Icinga2/Utils.pm            |  125 ++
 contrib/configconvert/README                      |   50 +
 contrib/configconvert/conf/.gitignore             |    1 +
 contrib/configconvert/icinga2-conv.conf           |   31 +
 contrib/configconvert/icinga2_convert_v1_v2.pl    |  152 ++
 9 files changed, 3514 insertions(+), 0 deletions(-)

Diff:   https://git.icinga.org/?p=icinga2.git;a=commitdiff;h=d050ee09f812bf1a912d35927cda1bb33ed3165b




More information about the icinga-checkins mailing list