~/2025/11/12/powershell-scom-alert-when-a-custom-event-log-source-goes-silent.md
PowerShell: SCOM – Alert When a Custom Event Log Source Goes Silent
--- author: Tom Lasswell date: read: 5 min in: [ps, scripts] tags: [powershell, scom] ---
$ grep -n '^#' post.md
SCOM is very good at alerting on an event that happens. Alerting on an event that stops happening takes more thought, and it's exactly the failure mode of a backup agent, a sync job, or any homegrown service that writes a heartbeat entry to the Application log on success and simply goes quiet on failure. Operations Manager does have a native Missing Event monitor type: Microsoft's management pack authoring guide describes it as detecting "an error state from an expected event not being detected in a particular time window", with the example of a nightly transfer that must log success between 2:00 and 3:00 AM. That works well for a job with a fixed schedule and a known event ID. It fits less well when the rule is "this source must log something at least every 90 minutes, on these twelve servers, each with a different source name and threshold", and you'd rather change a scheduled task parameter than author and seal another monitor.
For that case I run a small watchdog on the monitored server. It checks when the watched source last wrote any event, and if that's older than the threshold it writes an Error event under its own source. When the source starts logging again it writes an Information "recovered" event. A single SCOM unit monitor with Simple Event Detection and Windows Event Reset turns that pair into a health state that goes red and back to green on its own, with an alert that auto-resolves.
Requirements
- Windows PowerShell 5.1 on the monitored server. The script uses
New-EventLogandWrite-EventLog, and Microsoft removed the*-EventLogcmdlets from PowerShell 6 and later, so it won't run underpwsh. - Local administrator rights once, to register the alert source with
New-EventLogbefore the task first runs (see Usage). After that the script runs fine as a Scheduled Task under SYSTEM. It checks for its source by reading that one log's registry key rather than calling[System.Diagnostics.EventLog]::SourceExists(), which has to search every log including Security, and which Microsoft documents as failing under LocalSystem. - A SCOM agent on the server, reporting to a management group, and the unit monitor described under Usage, created in an unsealed management pack.
- A Scheduled Task that runs the script on an interval noticeably shorter than
MaxSilenceMinutes. - Optional, for checking alerts from a management server: the
OperationsManagerPowerShell module (installed with the Operations console).
Parameters
| Name | Type | Required | Description |
|---|---|---|---|
SourceName | String | Yes | Event source (provider name) of the application being watched, as it appears in the log. |
MaxSilenceMinutes | Int | Yes | How long the source can go without writing an event before it's considered silent. |
LogName | String | No | Log the watched source writes to, and the log the watchdog writes into. Defaults to Application. |
AlertSource | String | No | Event source the script registers and writes its own events under. Defaults to CustomSourceWatchdog. |
AlertEventId | Int | No | Event ID of the Error event written when the source is silent. Defaults to 9999. |
RecoveryEventId | Int | No | Event ID of the Information event written when a silent source starts logging again. Defaults to 9998. |
Usage
Check a backup agent that should log something at least every 90 minutes:
.\Test-EventSourceHeartbeat.ps1 -SourceName "ContosoBackupAgent" -MaxSilenceMinutes 90
Give a second application its own alert source and ID pair, so the two show up as separate monitors and alerts:
.\Test-EventSourceHeartbeat.ps1 -SourceName "NightlySyncJob" -MaxSilenceMinutes 120 -AlertSource "SyncJobWatchdog" -AlertEventId 9500 -RecoveryEventId 9501
Sample output when the source has gone silent, and on the next run after it recovers:
Last event from 'ContosoBackupAgent' was at 2025-11-11 22:14:07, 143.2 minutes ago.
Threshold of 90 minutes exceeded. Wrote event 9999 (Error) under source 'CustomSourceWatchdog'.
Last event from 'ContosoBackupAgent' was at 2025-11-12 00:41:55, 3.4 minutes ago.
Source recovered. Wrote event 9998 (Information) under source 'CustomSourceWatchdog'.
Register the alert source, then the Scheduled Task that runs as SYSTEM every 15 minutes (run once, elevated). Registering the source here rather than on the task's first run matters because Windows needs a moment to enable a new source before events can be written under it. -RepetitionDuration is set explicitly to ten years so the repetition doesn't stop:
New-EventLog -LogName "Application" -Source "CustomSourceWatchdog"
$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument '-NoProfile -ExecutionPolicy Bypass -File "C:\Scripts\Test-EventSourceHeartbeat.ps1" -SourceName "ContosoBackupAgent" -MaxSilenceMinutes 90'
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) -RepetitionInterval (New-TimeSpan -Minutes 15) -RepetitionDuration (New-TimeSpan -Days 3650)
$principal = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Register-ScheduledTask -TaskName "Heartbeat - ContosoBackupAgent" -Action $action -Trigger $trigger -Principal $principal -Description "Writes SCOM watchdog events when ContosoBackupAgent goes silent"
Create the monitor in the Operations console (once per alert source and ID pair):
- Authoring > Management Pack Objects > Monitors, right-click, Create a Monitor > Unit Monitor.
- Choose Windows Events > Simple Event Detection > Windows Event Reset, and select an unsealed management pack.
- Target a class that exists on the servers you're watching, for example Windows Server Operating System, and leave the monitor disabled if you'll enable it only for a group of servers with an override.
- Unhealthy event: log
Application, expression Event ID equals9999and Event Source equalsCustomSourceWatchdog. - Healthy event: log
Application, expression Event ID equals9998and Event Source equalsCustomSourceWatchdog. - Map the unhealthy state to Critical (or Warning), tick Generate alerts for this monitor, and keep the alert set to resolve automatically when the monitor returns to healthy.
Confirm the alert from a management server with the OperationsManager module:
Import-Module OperationsManager
Get-SCOMAlert -Criteria "ResolutionState != 255 and LastModified > '$((Get-Date).AddHours(-4))'" |
Where-Object { $_.Name -eq "<alert name from the monitor>" } |
Select-Object -Property Name, MonitoringObjectDisplayName, Severity, TimeRaised, ResolutionState
Script
<#
.SYNOPSIS
Writes SCOM-ready events when a custom event log source stops logging, and when it recovers.
.DESCRIPTION
Finds the most recent event written by a given event source and compares its timestamp against a
silence threshold. If the source has never logged, or its last event is older than the threshold,
the script writes an Error event (AlertEventId) under a separate alert source. If the source is
healthy and the watchdog's own last event was that Error, it writes an Information event
(RecoveryEventId) so a SCOM Windows Event Reset monitor returns to healthy. Intended to run on a
schedule on the monitored server under Windows PowerShell 5.1.
.PARAMETER SourceName
Event source (provider name) of the application being watched.
.PARAMETER MaxSilenceMinutes
How long the source can go without writing an event before it is considered silent.
.PARAMETER LogName
Log the watched source writes to, and the log the watchdog writes into. Defaults to Application.
.PARAMETER AlertSource
Event source the script registers and writes its own events under. Defaults to CustomSourceWatchdog.
.PARAMETER AlertEventId
Event ID of the Error event written when the source is silent. Defaults to 9999.
.PARAMETER RecoveryEventId
Event ID of the Information event written when a silent source starts logging again. Defaults to 9998.
.EXAMPLE
.\Test-EventSourceHeartbeat.ps1 -SourceName "ContosoBackupAgent" -MaxSilenceMinutes 90
.NOTES
Author : Thomas Lasswell (https://www.techcolumnist.com)
Version : 1.0 (2025-11-12)
Requires: Windows PowerShell 5.1, local administrator rights to register the alert event source
(register it once, elevated, before scheduling the script)
#>
#Requires -Version 5.1
#Requires -PSEdition Desktop
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[string]$SourceName,
[Parameter(Mandatory = $true)]
[ValidateRange(1, 10080)]
[int]$MaxSilenceMinutes,
[Parameter(Mandatory = $false)]
[string]$LogName = "Application",
[Parameter(Mandatory = $false)]
[string]$AlertSource = "CustomSourceWatchdog",
[Parameter(Mandatory = $false)]
[ValidateRange(1, 65535)]
[int]$AlertEventId = 9999,
[Parameter(Mandatory = $false)]
[ValidateRange(1, 65535)]
[int]$RecoveryEventId = 9998
)
if ($AlertEventId -eq $RecoveryEventId) {
throw "AlertEventId and RecoveryEventId must be different."
}
# Register the watchdog's own source if setup didn't. Test the one registry key instead of calling
# SourceExists, which searches every log (including Security) and is documented to fail under LocalSystem.
if (-not (Test-Path -Path "HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\$LogName\$AlertSource")) {
New-EventLog -LogName $LogName -Source $AlertSource
}
$lastEvent = Get-WinEvent -FilterHashtable @{ LogName = $LogName; ProviderName = $SourceName } -MaxEvents 1 -ErrorAction SilentlyContinue
$lastWatchdogEvent = Get-WinEvent -FilterHashtable @{ LogName = $LogName; ProviderName = $AlertSource } -MaxEvents 1 -ErrorAction SilentlyContinue
$now = Get-Date
if (-not $lastEvent) {
$message = "Source '$SourceName' has no events in the '$LogName' log at all. Either it has never run on $env:COMPUTERNAME, the log has rolled over since, or the source name is wrong."
Write-Warning $message
Write-EventLog -LogName $LogName -Source $AlertSource -EventId $AlertEventId -EntryType Error -Message $message
return
}
$silentMinutes = [math]::Round(($now - $lastEvent.TimeCreated).TotalMinutes, 1)
Write-Host "Last event from '$SourceName' was at $($lastEvent.TimeCreated.ToString('yyyy-MM-dd HH:mm:ss')), $silentMinutes minutes ago."
if ($silentMinutes -gt $MaxSilenceMinutes) {
$message = "Source '$SourceName' on $env:COMPUTERNAME has been silent for $silentMinutes minutes, exceeding the $MaxSilenceMinutes minute threshold. Last event: ID $($lastEvent.Id) at $($lastEvent.TimeCreated.ToString('yyyy-MM-dd HH:mm:ss'))."
Write-EventLog -LogName $LogName -Source $AlertSource -EventId $AlertEventId -EntryType Error -Message $message
Write-Host "Threshold of $MaxSilenceMinutes minutes exceeded. Wrote event $AlertEventId (Error) under source '$AlertSource'."
return
}
# Healthy. Only write a recovery event if the watchdog's last word was an alert, to keep the log quiet.
if ($lastWatchdogEvent -and $lastWatchdogEvent.Id -eq $AlertEventId) {
$message = "Source '$SourceName' on $env:COMPUTERNAME is logging again. Last event: ID $($lastEvent.Id) at $($lastEvent.TimeCreated.ToString('yyyy-MM-dd HH:mm:ss'))."
Write-EventLog -LogName $LogName -Source $AlertSource -EventId $RecoveryEventId -EntryType Information -Message $message
Write-Host "Source recovered. Wrote event $RecoveryEventId (Information) under source '$AlertSource'."
} else {
Write-Host "Within threshold. No event written."
}
Notes
- One alert source per watched application. The recovery logic reads the watchdog's own last event, so two watched applications sharing one
AlertSourcewould reset each other. Give each its ownAlertSourceand ID pair, and one monitor per pair, so alerts are distinguishable in the console instead of collapsing into one generic "something is quiet" alert. - Why a monitor and not a rule. An alert-generating rule on event 9999 raises an alert every time the event is written, which is every run while the source stays silent, and nothing closes those alerts when the source recovers. A unit monitor changes state once, raises one alert, and with Windows Event Reset returns to healthy on event 9998. The management pack authoring guide describes the three reset options: event reset, manual reset (a person must reset health in Health Explorer), and timer reset (automatic reset after a set time).
- When to use the native Missing Event monitor instead. If the application has a fixed schedule and logs a known success event, a Missing Event monitor needs no script on the server at all. The watchdog earns its place when thresholds differ per server or you want a rolling "nothing in N minutes" window rather than a scheduled one.
- Silence versus a rolled-over log. The script looks at what's in the log now. If the Application log is small and busy, the watched source's last event can be overwritten before the threshold is reached, and the watchdog reports "no events at all". Size the log so it holds more than
MaxSilenceMinutesof history. - Event IDs.
Write-EventLogaccepts IDs up to 65535. Stay clear of IDs the watched application itself uses, and remember that SCOM matches on source and ID, which is what keeps 9999 from one watchdog from tripping another's monitor. - Scheduling. Run the task at a fraction of the threshold. Every 15 minutes against a 90-minute threshold gives you an alert within 15 minutes of the threshold being crossed, and a recovery within 15 minutes of the source logging again.
- This deliberately doesn't touch the SCOM SDK on the monitored server. It writes plain Windows events and lets the agent that's already there, and already reporting to the management servers, do the rest.
Source
- Operations Manager management pack authoring: unit monitors (event detection and reset types)
- SCOM: monitor a specific Windows event (Operations console walkthrough)
- Write-EventLog (Windows PowerShell 5.1)
- Differences between Windows PowerShell 5.1 and PowerShell 7 (removed *-EventLog cmdlets)
- Get-SCOMAlert
- New-ScheduledTaskTrigger