Segfault in icingadb version 1.4.0

[smainz]

Describe the bug

This morning my icingadb stopped working (crashed)

This is the output of systemctl status icingadb.service

× icingadb.service - Icinga DB
     Loaded: loaded (/lib/systemd/system/icingadb.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2025-08-05 05:08:53 CEST; 15h ago
   Duration: 2h 27min 57.293s
    Process: 936 ExecStart=/usr/sbin/icingadb --config /etc/icingadb/config.yml (code=exited, status=2)
   Main PID: 936 (code=exited, status=2)
        CPU: 1min 47.524s

Aug 05 05:08:53 monitoring icingadb[936]: r11    0x246
Aug 05 05:08:53 monitoring icingadb[936]: r12    0x7fe76cff8630
Aug 05 05:08:53 monitoring icingadb[936]: r13    0xc000d6be6a
Aug 05 05:08:53 monitoring icingadb[936]: r14    0xc00047aa80
Aug 05 05:08:53 monitoring icingadb[936]: r15    0x0
Aug 05 05:08:53 monitoring icingadb[936]: rip    0x44991d
Aug 05 05:08:53 monitoring icingadb[936]: rflags 0x10212
Aug 05 05:08:53 monitoring icingadb[936]: cs     0x33
Aug 05 05:08:53 monitoring icingadb[936]: fs     0x0
Aug 05 05:08:53 monitoring icingadb[936]: gs     0x0

This is the output of journalctl -u icingadb.service --since "2025-08-05 05:00:00"

Aug 05 05:00:57 monitoring icingadb[936]: high-availability: Continuing being the active instance
Aug 05 05:05:57 monitoring icingadb[936]: high-availability: Continuing being the active instance
Aug 05 05:08:53 monitoring icingadb[936]: SIGSEGV: segmentation violation
Aug 05 05:08:53 monitoring icingadb[936]: PC=0x44991d m=10 sigcode=1 addr=0x1a7e0d0
Aug 05 05:08:53 monitoring icingadb[936]: goroutine 0 gp=0xc00047aa80 m=10 mp=0xc000080808 [idle]:
Aug 05 05:08:53 monitoring icingadb[936]: runtime.findRunnable()
Aug 05 05:08:53 monitoring icingadb[936]:         runtime/proc.go:3282 +0x5dd fp=0x7fe76cff8da8 sp=0x7fe76cff8c30 pc=0x44991d
Aug 05 05:08:53 monitoring icingadb[936]: runtime: g 0: unexpected return pc for runtime.findRunnable called from 0x7fe76cff>
Aug 05 05:08:53 monitoring icingadb[936]: stack: frame={sp:0x7fe76cff8c30, fp:0x7fe76cff8da8} stack=[0x7fe76c7f9260,0x7fe76c>
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b30:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b40:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b50:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b60:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b70:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b80:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8b90:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8ba0:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8bb0:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8bc0:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8bd0:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8be0:  0x0000000000000000  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8bf0:  0x000008139c1074b8  0x00007fe76cff8c20
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c00:  0x0000000000419011 <runtime.unlock2+0x0000000000000051>  0x00>
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c10:  0x000008139ef27d71  0x000000c00047aa80
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c20:  0x00007fe76cff8da0  0x000000000044991d <runtime.findRunnable+>
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c30: <0x0000000000449745 <runtime.findRunnable+0x0000000000000405> >
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c40:  0x000000000042d079 <runtime.(*scavengeIndex).alloc+0x00000000>
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c50:  0x0000000002030007  0x000080c001d3e000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c60:  0x020000af2c200000  0x00007f01000009a3
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c70:  0x00007fe76cff8d08  0x0000000000000000
Aug 05 05:08:53 monitoring icingadb[936]: 0x00007fe76cff8c80:  0x00000000012d3268  0x0000000002030007

To Reproduce

I have no clue, it happened a few times before, but only this time I took the time to investigate.

Expected behavior

Just don't crash

Your Environment

• Icinga DB version:

  icingadb --version
  Icinga DB version: v1.4.0
  
  Build information:
    Go version: go1.24.4 (linux, amd64)
    Git commit: c271de72383e3b5a6fa1e78cbd9f90d74fff6d2b
  
  System information:
    Platform: Debian GNU/Linux
    Platform version: 12 (bookworm)
  

• Icinga 2 version: icinga2 - The Icinga 2 network monitoring daemon (version: r2.15.0-1)

• Operating System and version: Debian Bookworn (running as VM on Proxmox)

Additional context

Installed from deb [signed-by=/usr/share/keyrings/icinga-archive-keyring.gpg] https://packages.icinga.com/debian icinga-bookworm main

[yhabteab]

SIGSEGV: segmentation violation
Aug 05 05:08:53 monitoring icingadb[936]: PC=0x44991d m=10 sigcode=1 addr=0x1a7e0d0
Aug 05 05:08:53 monitoring icingadb[936]: goroutine 0 gp=0xc00047aa80 m=10 mp=0xc000080808 [idle]:
Aug 05 05:08:53 monitoring icingadb[936]: runtime.findRunnable()
Aug 05 05:08:53 monitoring icingadb[936]:         runtime/proc.go:3282 +0x5dd fp=0x7fe76cff8da8 sp=0x7fe76cff8c30 pc=0x44991d
Aug 05 05:08:53 monitoring icingadb[936]: runtime: g 0: unexpected return pc for runtime.findRunnable called from 0x7fe76cff>

Oh no, this doesn't look good at all. It seems like either we somehow (to be specific, the dependencies we use, since we don't use any unsafe code) corrupted the memory, or even though it's the least likely, but still possible, a Go runtime bug.

$ echo "0x44991d" | go1.24.4 tool addr2line /usr/sbin/icingadb
runtime.findRunnable
runtime/proc.go:3282
$ echo "0x419011" | go1.24.4 tool addr2line /usr/sbin/icingadb
runtime.unlock2
runtime/lock_spinbit.go:272
$ echo "0x449745" | go1.24.4 tool addr2line /usr/sbin/icingadb
runtime.findRunnable
runtime/proc.go:3390
$ echo "0x42d079" | go1.24.4 tool addr2line /usr/sbin/icingadb
runtime.(*scavengeIndex).alloc
runtime/mgcscavenge.go:1140

Are you sure that Icinga DB has started to crash like this after the upgrade to 1.4.0? Also, does it crash regularly, or is this just once in a while? You mentioned that this happens a few times before but were the crashes always like this, or did you have different crash logs? The dump you provided above seems to be also truncated, can you please provide the full crash log if possible?

Were you also able to identify any specific patterns when this happens? Does Icinga DB logs the same messages each time before the crash, or is it random?

[julianbrost]

The dump you provided above seems to be also truncated, can you please provide the full crash log if possible?

Hint: You might want to run journalctl with the additional --no-pager flag. Otherwise it'll automatically start a pager which is nice for interactive use, but not for copying from it.

Though from what I can see, this looks unlikely to be caused by the Icinga DB code itself, I believe one of these causes is more likely:

• Bug in the Go compiler or runtime
• Broken use of unsafe or cgo in some dependency
• Hardware fault, like broken memory

[smainz]

The last two or three times I noticed, icingadb was stopped, I just restarted it and did not care to investigate, so I can not say if the problem was the same. It did not happen too often, so I only started to investigate now.

In the mean time I ran a memtester with no fault

The complete log is 11.426 lines long.

icingadb.log

Any ideas what I can do to help / analyze?

[yhabteab]

Apologies for the delay!

Well, the backtraces you provided are a lot and very verbose but still I wasn't able to post factum identify the offending goroutine that corrupted the memory. If you can manually build and deploy Icinga DB on that machine, you could try building it with the race detector enabled, and see if it catches any race conditions that would otherwise cause such disastrous consequences.

Scanning the Go issue tracker for similar issues, I found this one which seems to describe a race condition in the Go runtime and the Garbage Collector (GC). I'm saying this because in the crash dumps you provided, the scavenger subsystem of the GC is also involved, but I can't tell you for sure if this is the same issue. However, you can try to upgrade your Go version (if not to Go 1.25) at the very least to Go 1.24.5 and rebuild the Icinga DB binary from source and see if the issue goes away, because the referenced issue was backported to Go 1.24.5.

[smainz]

Sorry for answering late, I have been on holiday and away from keyboard.

I can not reproduce the error and did not have any crashes since then. So probably this was one of the mysteries we never can solve. Maybe we should close this issue.