[icinga-users] 2.4.1 crashing :(

Zachary McGibbon zachary.mcgibbon at gmail.com
Fri Nov 27 14:39:58 CET 2015


Michael, I got a crash finally and got a backtrace.  If ever you want to do
some troubleshooting with me we could always do a Skype call with desktop
sharing or something like that, always glad to help :)

Detaching after fork from child process 27787.

Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffef38e700 (LWP 26406)]
0x00000038ef60e2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64
libedit-2.11-4.20080712cvs.1.el6.x86_64 libgcc-4.4.7-16.el6.x86_64
libicu-4.2.1-12.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
libstdc++-4.4.7-16.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
ncurses-libs-5.7-4.20090207.el6.x86_64
nss-softokn-freebl-3.14.3-23.el6_7.x86_64 openssl-1.0.1e-42.el6.x86_64
zlib-1.2.3-29.el6.x86_64
(gdb)
(gdb) bt full
#0  0x00000038ef60e2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000038ef609588 in _L_lock_854 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00000038ef609457 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007ffff7d5f498 in lock (this=0x7fffef38c7f0) at
/usr/include/boost153/boost/thread/pthread/mutex.hpp:67
        res = <value optimized out>
#4  boost::unique_lock<boost::mutex>::lock (this=0x7fffef38c7f0) at
/usr/include/boost153/boost/thread/lock_types.hpp:344
No locals.
#5  0x00007ffff7d2fbf5 in unique_lock (this=0x7fff2f47d0a0, callback=...)
at /usr/include/boost153/boost/thread/lock_types.hpp:124
No locals.
#6  icinga::Process::Run(const boost::function<void(const
icinga::ProcessResult&)> &) (this=0x7fff2f47d0a0, callback=...) at
../base/process.cpp:661
        lock = {m = 0x7ffff7ff0220, is_locked = false}
        fds = {16, 18}
        __PRETTY_FUNCTION__ = "void icinga::Process::Run(const
boost::function<void(const icinga::ProcessResult&)>&)"
        envc = <value optimized out>
        tid = 0
        use_vfork = {m_Value = {which_ = 3, storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{buf = "\b\231c\000\000\000\000",
                  align_ = {<No data fields>}}}, static size = <optimized
out>, static alignment = <optimized out>}}}
#7  0x00007ffff5bda4ee in (anonymous
namespace)::PluginUtility::ExecuteCommand(const (anonymous
namespace)::Command::Ptr &, const (anonymous namespace)::Checkable::Ptr &,
const (anonymous namespace)::CheckResult::Ptr &, const (anonymous
namespace)::MacroProcessor::ResolverList &, const (anonymous
namespace)::Dictionary::Ptr &, bool, const boost::function<void(const
icinga::Value&, const icinga::ProcessResult&)> &) (commandObj=...,
checkable=Unhandled dwarf expression opcode 0xf3
) at ../icinga/pluginutility.cpp:92
        raw_arguments = {px = 0x7fffd0009810}
        command = {m_Value = {which_ = 4, storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{buf = "P\363$\021\377\177\000",
                  align_ = {<No data fields>}}}, static size = <optimized
out>, static alignment = <optimized out>}}}
        env = {px = 0x0}
        process = {px = 0x7fff2f47d0a0}
        raw_command = {m_Value = {which_ = 4, storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{buf = "\340m\000\320\377\177\000",
                  align_ = {<No data fields>}}}, static size = <optimized
out>, static alignment = <optimized out>}}}
        envMacros = {px = 0x7fff46c7bc00}
#8  0x00007ffff55b1b12 in icinga::PluginCheckTask::ScriptFunc
(checkable=..., cr=Unhandled dwarf expression opcode 0xf3
) at ../methods/pluginchecktask.cpp:57
        commandObj = {px = 0x7fffd0008db0}
        host = {px = 0x7fffc41a8cd0}
        service = <value optimized out>
        resolvers = std::vector of length 4, capacity 4 = {{first = {static
NPos = 18446744073709551615, m_Data = "service"}, second = {px =
0x7fffc8e27ec0}}, {first = {
              static NPos = 18446744073709551615, m_Data = "host"}, second
= {px = 0x7fffc41a8cd0}}, {first = {static NPos = 18446744073709551615,
m_Data = "command"},
            second = {px = 0x7fffd0008db0}}, {first = {static NPos =
18446744073709551615, m_Data = "icinga"}, second = {px = 0x7fffc839c920}}}
#9  0x00007ffff55bb7d6 in
icinga::FunctionWrapperV<boost::intrusive_ptr<icinga::Checkable> const&,
boost::intrusive_ptr<icinga::CheckResult> const&,
boost::intrusive_ptr<icinga::Dictionary> const&, bool> (
    function=0x7ffff55b1560
<icinga::PluginCheckTask::ScriptFunc(icinga::Checkable::Ptr const&,
icinga::CheckResult::Ptr const&, icinga::Dictionary::Ptr const&, bool)>,
    arguments=std::vector of length 4, capacity 4 = {...}) at
../base/functionwrapper.hpp:171
        __PRETTY_FUNCTION__ = "icinga::Value icinga::FunctionWrapperV(void
(*)(T0, T1, T2, T3), const std::vector<icinga::Value>&) [with T0 = const
boost::intrusive_ptr<icinga::Checkable>&; T1 = const
boost::intrusive_ptr<icinga::C"...
#10 0x00007ffff55b8add in operator()<icinga::Value, icinga::Value (*)(void
(*)(const boost::intrusive_ptr<icinga::Checkable>&, const
boost::intrusive_ptr<icinga::CheckResult>&, ---Type <return> to continue,
or q <return> to quit---
const boost::intrusive_ptr<icinga::Dictionary>&, bool), const
std::vector<icinga::Value>&), boost::_bi::list1<const
std::vector<icinga::Value>&> > (function_obj_ptr=Unhandled dwarf expression
opcode 0xf3
)
    at /usr/include/boost153/boost/bind/bind.hpp:303
No locals.
#11 operator()<std::vector<icinga::Value> > (function_obj_ptr=Unhandled
dwarf expression opcode 0xf3
) at /usr/include/boost153/boost/bind/bind_template.hpp:47
No locals.
#12
boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<icinga::Value,
icinga::Value (*)(void (*)(boost::intrusive_ptr<icinga::Checkable> const&,
boost::intrusive_ptr<icinga::CheckResult> const&,
boost::intrusive_ptr<icinga::Dictionary> const&, bool),
std::vector<icinga::Value, std::allocator<icinga::Value> > const&),
boost::_bi::list2<boost::_bi::value<void
(*)(boost::intrusive_ptr<icinga::Checkable> const&,
boost::intrusive_ptr<icinga::CheckResult> const&,
boost::intrusive_ptr<icinga::Dictionary> const&, bool)>, boost::arg<1> > >,
icinga::Value, std::vector<icinga::Value, std::allocator<icinga::Value> >
const&>::invoke (function_obj_ptr=Unhandled dwarf expression opcode 0xf3
)
    at /usr/include/boost153/boost/function/function_template.hpp:132
        f = <error reading variable f (Unhandled dwarf expression opcode
0xf3)>
#13 0x00007ffff7cfdfdd in operator() (this=Unhandled dwarf expression
opcode 0xf3
) at /usr/include/boost153/boost/function/function_template.hpp:767
No locals.
#14 icinga::Function::Invoke (this=Unhandled dwarf expression opcode 0xf3
) at ../base/function.cpp:34
No locals.
#15 0x00007ffff5bba92f in (anonymous namespace)::CheckCommand::Execute
(this=0x7fffd0008db0, checkable=Unhandled dwarf expression opcode 0xf3
) at ../icinga/checkcommand.cpp:36
        arguments = std::vector of length 4, capacity 4 = {{m_Value =
{which_ = 4, storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{
                    buf = "\300~\342\310\377\177\000", align_ = {<No data
fields>}}}, static size = <optimized out>, static alignment = <optimized
out>}}}, {m_Value = {
              which_ = 4, storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{buf = "`BG\030\377\177\000", align_ = {<No data fields>}}},
                static size = <optimized out>, static alignment =
<optimized out>}}}, {m_Value = {which_ = 0,
              storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{buf = " \311\071\310\377\177\000", align_ = {<No data fields>}}},
                static size = <optimized out>, static alignment =
<optimized out>}}}, {m_Value = {which_ = 2,
              storage_ =
{<boost::detail::aligned_storage::aligned_storage_imp<8ul, 8ul>> = {data_ =
{buf = "\000_ping\"", align_ = {<No data fields>}}},
                static size = <optimized out>, static alignment =
<optimized out>}}}}
#16 0x00007ffff5c11176 in (anonymous namespace)::Checkable::ExecuteCheck
(this=0x7fffc8e27ec0) at ../icinga/checkable-check.cpp:420
        currentContextFrame = {<No data fields>}
        reachable = <value optimized out>
        before_check = <value optimized out>
        local = <value optimized out>
        scheduled_start = <value optimized out>
        cr = {px = 0x7fff18474260}
        endpoint = {px = 0x0}
#17 0x00007ffff538075f in icinga::CheckerComponent::ExecuteCheckHelper
(this=0x7fffcc01f130, checkable=...) at ../checker/checkercomponent.cpp:200
No locals.
#18 0x00007ffff7d4523c in operator() (this=0x7ffff7fef830, queue=...) at
/usr/include/boost153/boost/function/function_template.hpp:767
No locals.
#19 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x7ffff7fef830,
queue=...) at ../base/threadpool.cpp:138
        wi = {Callback = {<boost::function0<void>> =
{<boost::function_base> = {vtable = 0x7ffff559bbc0, functor = {obj_ptr =
0x7fff39760eb0, type = {type = 0x7fff39760eb0,
                    const_qualified = false, volatile_qualified = false},
func_ptr = 0x7fff39760eb0, bound_memfunc_ptr = {memfunc_ptr = (void
(boost::detail::function::X::*)(
    boost::detail::function::X *, int)) 0x7fff39760eb0, obj_ptr =
0x7fffcde8ff00}, obj_ref = {obj_ptr = 0x7fff39760eb0, is_const_qualified =
false,
                    is_volatile_qualified = false}, data = -80 '\260'}},
static args = <optimized out>, static arity = <optimized out>}, <No data
fields>},
          Timestamp = 1448613562.156776}
        st = 1448613619.7012911
        et = <value optimized out>
        latency = <value optimized out>
---Type <return> to continue, or q <return> to quit---
        idbuf = <incomplete type>
        lock = {m = 0x7ffff559bbc0, is_locked = 176}
#20 0x0000003e0040c5c3 in boost::(anonymous namespace)::thread_proxy
(param=<value optimized out>) at libs/thread/src/pthread/thread.cpp:165
        thread_info = {px = 0x7fffd407c950, pn = {pi_ = 0x7fffd40bb030}}
        lock = <value optimized out>
#21 0x00000038ef607a51 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#22 0x00000038ef2e893d in clone () from /lib64/libc.so.6
No symbol table info available.


On Thu, Nov 26, 2015 at 2:07 PM, Zachary McGibbon <
zachary.mcgibbon at gmail.com> wrote:

> That was all I got.. I tried kill but then I didn't get anything... will
> try and run it again and see if it'll crash.  I haven't re-enabled the
> scheduled downtimes so I'm quite sure it isn't that causing the issue.
>
> On Thu, Nov 26, 2015 at 2:01 PM, Michael Friedrich <
> Michael.Friedrich at netways.de> wrote:
>
>>
>> > On 26 Nov 2015, at 19:53, Zachary McGibbon <zachary.mcgibbon at gmail.com>
>> wrote:
>> >
>> > I've been having intermitent crashing problems over the past few days,
>> and upgraded to 2.4.1 today and still crashing…
>>
>> I don’t see any crash in your output, any more details?
>>
>>
>> > running gdb but not getting the results I want, suggestions?  My system
>> won't install the missing debug info packages..  Last output from gdb:
>>
>> Generate a backtrace and pipe it into a log file, attached here.
>>
>>
>> http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/development#development-debug-gdb-backtrace
>>
>> Kind regards,
>> Michael
>>
>> >
>> > [2015-11-26 13:10:57 -0500] notice/ApiListener: Relaying
>> 'event::CheckResult' message
>> > [2015-11-26 13:10:57 -0500] notice/ApiListener: Relaying
>> 'event::CheckResult' message
>> > [2015-11-26 13:10:57 -0500] notice/ApiListener: Relaying
>> 'event::CheckResult' message
>> > [2015-11-26 13:10:57 -0500] notice/ThreadPool: Thread pool; current:
>> 16; adjustment: 2
>> > [2015-11-26 13:10:57 -0500] notice/ThreadPool: Thread pool; current:
>> 14; adjustment: -2
>> > [2015-11-26 13:10:57 -0500] debug/ThreadPool: Killing worker thread.
>> > [2015-11-26 13:10:57 -0500] debug/ThreadPool: Killing worker thread.
>> > Cannot get thread event message: debugger service failed
>> > Missing separate debuginfos, use: debuginfo-install
>> glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64
>> krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64
>> libedit-2.11-4.20080712cvs.1.el6.x86_64 libgcc-4.4.7-16.el6.x86_64
>> libicu-4.2.1-12.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>> libstdc++-4.4.7-16.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>> ncurses-libs-5.7-4.20090207.el6.x86_64
>> nss-softokn-freebl-3.14.3-23.el6_7.x86_64 openssl-1.0.1e-42.el6.x86_64
>> zlib-1.2.3-29.el6.x86_64
>> > (gdb) bt full
>> > Target is executing.
>> >
>> > _______________________________________________
>> > icinga-users mailing list
>> > icinga-users at lists.icinga.org
>> > https://lists.icinga.org/mailman/listinfo/icinga-users
>>
>>
>> --
>> Michael Friedrich, DI (FH)
>> Senior Developer
>>
>> NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg
>> Tel: +49 911 92885-0 | Fax: +49 911 92885-77
>> GF: Julian Hein, Bernd Erk | AG Nuernberg HRB18461
>> http://www.netways.de | Michael.Friedrich at netways.de
>>
>> ** OSDC 2016 - April - netways.de/osdc **
>> ** OSBConf 2016 - September - osbconf.org **
>> _______________________________________________
>> icinga-users mailing list
>> icinga-users at lists.icinga.org
>> https://lists.icinga.org/mailman/listinfo/icinga-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.icinga.org/pipermail/icinga-users/attachments/20151127/4970a482/attachment-0001.html>


More information about the icinga-users mailing list