Recovery notification outside time period still lost #10575
Description
Hello Icinga-Team :)
Describe the bug
We just stumbled upon this issue again, though it should have been fixed with #10187
We have a check that recovered while outside it's notification period (04.10. 23:58) and then should have sent out the recovery notification on 06.10. 08:00.
Time period:
object TimePeriod "8x5" {
import "legacy-timeperiod"
ranges = {
"friday" = "08:00-17:00"
"monday" = "08:00-17:00"
"thursday" = "08:00-17:00"
"tuesday" = "08:00-17:00"
"wednesday" = "08:00-17:00"
}
}
Service history:
User object:
object User "ticketsystem" {
import "generic-user-template-problem_down-critical"
states = [ Critical, Down, OK, Up ]
types = [ Custom, Problem, Recovery ]
vars.fallback_email = "..."
}
Until I discovered the mentioned issue I wasn't sure if this is "works as designed", but apparently it shouldn't be.
This is the debug log from around the time the notification was suppressed
[2025-10-03 23:58:24 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-03 23:58:24 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!pending_updates!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-03 23:58:25 +0200] debug/ApiListener: Not checking connection to Zone 'abcd-dc02' because it's not in the same zone, a parent or a child zone.
[2025-10-03 23:58:25 +0200] debug/ApiListener: Not checking connection to Zone 'abcd-dc02' because it's not in the same zone, a parent or a child zone.
[2025-10-03 23:58:25 +0200] debug/ApiListener: Not checking connection to Zone 'abcd-dc02' because it's not in the same zone, a parent or a child zone.
[2025-10-03 23:58:27 +0200] debug/ApiListener: Not checking connection to Zone 'abcd-dc02' because it's not in the same zone, a parent or a child zone.
[2025-10-03 23:58:29 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-03 23:58:29 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!pending_updates!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-03 23:58:29 +0200] information/Checkable: Acknowledgement cleared for checkable 'abcd-dc02!commvault-backup'.
[2025-10-03 23:58:29 +0200] notice/ApiListener: Received config delete for object: {"name":"abcd-dc02!commvault-backup!37e51972-af95-480d-b266-faf9c7be6fa2","type":"Comment","version":1759471212.819265}
[2025-10-03 23:58:29 +0200] notice/ApiListener: Discarding 'config delete object' message from 'abcd-lm1-fra' (endpoint: 'abcd-lm1-fra', zone: 'abcd-fra') for object 'abcd-dc02!commvault-backup!37e51972-af95-480d-b266-faf9c7be6fa2' of type 'Comment'. Sender is in a child zone.
[2025-10-03 23:58:29 +0200] notice/Comment: Removed comment 'abcd-dc02!commvault-backup!37e51972-af95-480d-b266-faf9c7be6fa2' from object 'abcd-dc02!commvault-backup'.
[2025-10-03 23:58:29 +0200] information/ConfigObjectUtility: Deleted object 'abcd-dc02!commvault-backup!37e51972-af95-480d-b266-faf9c7be6fa2' of type 'Comment'.
[2025-10-03 23:58:29 +0200] notice/Checkable: State Change: Checkable 'abcd-dc02!commvault-backup' hard state change from CRITICAL to OK detected.
[2025-10-03 23:58:29 +0200] information/Checkable: Checkable 'abcd-dc02!commvault-backup' has 1 notification(s). Checking filters for type 'Recovery', sends will be logged.
[2025-10-03 23:58:29 +0200] notice/Notification: Attempting to send notifications of type 'Recovery' for notification object 'abcd-dc02!commvault-backup!notification-jira-service_critical'.
[2025-10-03 23:58:29 +0200] notice/Notification: Not sending notifications for notification object 'abcd-dc02!commvault-backup!notification-jira-service_critical': not in timeperiod '8x5'
and here the log from when the notification was expected
[2025-10-06 07:59:32 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-dc02!commvault-logbackup!Commvault-Maintenance Services"
[2025-10-06 07:59:32 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-dc02!commvault-backup!Commvault-Maintenance Services"
[2025-10-06 07:59:37 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 07:59:42 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 07:59:47 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 07:59:52 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 07:59:58 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:03 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:09 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:14 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:19 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:24 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:29 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:35 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-10-06 08:00:35 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-dc02!commvault-logbackup!Commvault-Maintenance Services"
[2025-10-06 08:00:35 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-dc02!commvault-backup!Commvault-Maintenance Services"
[2025-10-06 08:00:52 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"85f19ab7-7ef1-4ce7-bd6c-eb2302e328f0\" ignore_on_error {\n\tauthor = \"Icinga Admin DSL\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"Scheduled Update\"\n\tconfig_owner = \"abcd-dc02!commvault-backup!scheduled_update\"\n\tconfig_owner_hash = \"1cea808495b94018c496a5b71c0ff4d6202e717eb4017f5e38c9229287418dcd\"\n\tduration = 3600.000000\n\tend_time = 1760904000.000000\n\tentry_time = 1759269882.441912\n\tfixed = false\n\thost_name = \"abcd-dc02\"\n\tparent = \"\"\n\tscheduled_by = \"abcd-dc02!commvault-backup!scheduled_update\"\n\tservice_name = \"commvault-backup\"\n\tstart_time = 1760896800.000000\n\ttriggered_by = \"\"\n\tversion = 1759269882.441952\n\tzone = \"abcd-fra\"\n}\n","modified_attributes":{},"name":"abcd-dc02!commvault-backup!85f19ab7-7ef1-4ce7-bd6c-eb2302e328f0","original_attributes":[],"type":"Downtime","version":1759269882.441952,"zone":"abcd-fra"}
[2025-10-06 08:00:52 +0200] notice/ApiListener: Discarding 'config update object' message from 'abcd-lm1-fra' (endpoint: 'abcd-lm1-fra', zone: 'abcd-fra') for object 'abcd-dc02!commvault-backup!85f19ab7-7ef1-4ce7-bd6c-eb2302e328f0' of type 'Downtime'. Sender is in a child zone.
[2025-10-06 08:00:52 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"52e32f9d-e102-4a4c-9048-1ea3e2535548\" ignore_on_error {\n\tauthor = \"Icinga Admin DSL\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"Scheduled Update\"\n\tconfig_owner = \"abcd-dc02!commvault-logbackup!scheduled_update\"\n\tconfig_owner_hash = \"1cea808495b94018c496a5b71c0ff4d6202e717eb4017f5e38c9229287418dcd\"\n\tduration = 3600.000000\n\tend_time = 1760904000.000000\n\tentry_time = 1759269882.342413\n\tfixed = false\n\thost_name = \"abcd-dc02\"\n\tparent = \"\"\n\tscheduled_by = \"abcd-dc02!commvault-logbackup!scheduled_update\"\n\tservice_name = \"commvault-logbackup\"\n\tstart_time = 1760896800.000000\n\ttriggered_by = \"\"\n\tversion = 1759269882.342464\n\tzone = \"abcd-fra\"\n}\n","modified_attributes":{},"name":"abcd-dc02!commvault-logbackup!52e32f9d-e102-4a4c-9048-1ea3e2535548","original_attributes":[],"type":"Downtime","version":1759269882.342464,"zone":"abcd-fra"}
[2025-10-06 08:00:52 +0200] notice/ApiListener: Discarding 'config update object' message from 'abcd-lm1-fra' (endpoint: 'abcd-lm1-fra', zone: 'abcd-fra') for object 'abcd-dc02!commvault-logbackup!52e32f9d-e102-4a4c-9048-1ea3e2535548' of type 'Downtime'. Sender is in a child zone.
[2025-10-06 08:00:55 +0200] notice/NotificationComponent: Reminder notification 'abcd-dc02!commvault-backup!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
Not sure about the Creating Downtime messages, because that is an old ScheduledDowntime object with an specific date
apply ScheduledDowntime "Commvault-Maintenance Services" to Service {
author = "abcd"
comment = "Downtime für Commvault-Updates oder andere Commvault Wartungen.
fixed = true
assign where match("commvault*", service.name)
ranges = {
"2025-08-25" = "08:00-17:00"
}
}
But as the screenshot shows there was no downtime entered and imo the notification should have been sent out at 08:00 on 06.10.2025
The question is, why wasn't the notification triggered?
Your Environment
Include as many relevant details about the environment you experienced the problem in
- Version used (
icinga2 --version): 2.15.0 - Operating System and version: rhel8.10
- Single master instance with 90+ connected zones.
- Host that the notification is missing for belongs to a satellite zone.
Best regards
Metadata
Metadata
Assignees
Labels
No labels