tom@techcolumnist:~$

~/2026/07/22/infrastructure-what-a-real-disaster-recovery-test-actually-reveals.md

Infrastructure: What a Real Disaster Recovery Test Actually Reveals

---
author: 
date: 
read: 8 min
in:   [engineering, strategy]
tags: [backup, hyper-v]
---

$ grep -n '^#' post.md

A disaster recovery plan that has only ever been reviewed on paper is a hypothesis, not a capability. I've sat through plenty of tabletop exercises where everyone agreed the runbook made sense, the RTOs looked achievable, and the plan got signed off for another year — and then watched a real, full failover to the DR site find a dozen things the tabletop never could, because a tabletop only tests whether the plan reads well, not whether the systems actually do what the plan assumes.

Tabletop Exercises Don't Find These

Walking through a runbook out loud surfaces logic gaps — a missing step, an unclear owner, a sequencing question nobody had thought about. What it can't surface is anything that only breaks when bits actually move: a backup job that's been silently failing for a service nobody checks, a replication link that's been running behind for months without alerting because the alert threshold was set once and never revisited, or a script that references a path that stopped existing after a server rename. None of that shows up until someone actually tries to bring the environment up somewhere else.

The good news is that the tooling most of us already own was built for exactly this. Hyper-V Replica has a test failover that creates a temporary copy of the replica VM on the DR host, isolated from the production network, while the primary keeps running and replication carries on untouched. Azure Site Recovery has the same idea at recovery-plan scale: you pick a recovery point (latest processed, latest app-consistent, and so on), pick an isolated virtual network, and clean up the test VMs when you're done. Neither costs you downtime, which removes the usual excuse for only ever doing the tabletop.

The Dependencies Nobody Documented

Every DR plan I've reviewed has a failover group — the list of systems that get included in the test or the real event. Every real test I've run has found at least one application that depends on something outside that group. An app server fails over cleanly and then can't reach the license server, the internal API it calls for lookups, or a file share that was never in scope because nobody thought of it as "infrastructure." These dependencies are invisible in an architecture diagram that only shows what a team explicitly built, because most of them accumulated informally after the diagram was drawn. A real failover is the only reliable way to find them, because it's the only exercise that actually tries to use the dependency and fails when it isn't there.

That's why the test script below doesn't just check that VMs boot. It reads a plan file that lists, for each VM, the services that must be running and the TCP endpoints it must reach, and it tests those from inside the recovered guest. The plan file becomes the dependency documentation, and every failed endpoint check is either a missing VM in the failover group or a hard-coded address that doesn't exist at the DR site.

Licensing and Identity Break First

In the DR tests I've run, two categories of failure have come up more often than anything else, and neither is a capacity problem. Licensing tied to a hardware ID, a MAC address, or a specific IP will refuse to activate on DR hardware that doesn't match, and the fix is rarely fast if it requires a vendor support call mid-test. Identity is the other one: service accounts, Kerberos delegation, and certificate trust chains are full of assumptions about which domain controller, which site, or which network path is available, and a DR site that's a slower or differently-configured replica of production identity infrastructure surfaces authentication failures that never show up in normal operation.

Microsoft's Site Recovery guidance for Active Directory is a useful model even outside Azure. Test failover happens in a network isolated from production; before any application fails over, a domain controller and DNS server have to exist in that isolated network, most easily by test-failing-over a replicated DC first. That DC should be a global catalog and hold the FSMO roles the test needs, or those roles have to be seized afterwards. The isolated network should reuse the production address range and must not be connected back to production. On Hyper-V Replica, a test VM (named after the original with - Test appended) isn't connected to any network by default, which is safe but means no dependency can be tested. The equivalent of Site Recovery's isolated network is an internal or private virtual switch on the Replica server, assigned to each replicated adapter with Set-VMNetworkAdapter -TestReplicaSwitchName, plus a DC first in the plan file so it is up before anything that authenticates against it.

Recovery Time vs Recovery Confidence

The number everyone tracks is recovery time — how long the failover took against the target RTO. The number that actually matters more is recovery confidence: how much of the failover required a human improvising a fix in real time versus following the documented steps. A recovery that hit its RTO because three people quietly worked around three undocumented problems isn't a passing test; it's a test that got lucky with who happened to be in the room.

Automation doesn't create confidence by itself, but it measures it honestly. A script records the same timings and checks every run, so the only way the result improves is if the environment did. And the recovery point matters as much as the recovery time: Hyper-V Replica sends changes every 30 seconds, 5 minutes or 15 minutes, keeps up to 24 hourly recovery points if you configure them, and only produces application-consistent points when you schedule VSS snapshots. A test that fails over to a crash-consistent point on a database server is testing something different from what the business thinks it has.

The Test Failover Script

This runs on the Hyper-V Replica server that holds the replica VMs (PowerShell Direct only reaches VMs running on the local host). For each VM in the plan, in order, it checks replication health with Get-VMReplication, creates a test VM with Start-VMFailover -AsTest, starts it, waits until the guest answers over PowerShell Direct, gives services time to settle, then checks the listed services and endpoints from inside the guest. Test VMs stay running until every VM has been checked, because later VMs depend on earlier ones, and then Stop-VMFailover deletes them all, including when the run stops on an error. The results go to a CSV you can attach to the DR test record.

The plan file is a PowerShell data file, ordered the way the runbook boots things:

powershell
@{
    VMs = @(
        @{ Name = 'dc-dr01'; Services = @('NTDS', 'DNS', 'Netlogon'); Endpoints = @() }
        @{ Name = 'sql01'; Services = @('MSSQLSERVER', 'SQLSERVERAGENT'); Endpoints = @('dc-dr01:389', 'dc-dr01:88') }
        @{ Name = 'lic01'; Services = @('<license-service-name>'); Endpoints = @('dc-dr01:389') }
        @{ Name = 'app01'; Services = @('W3SVC'); Endpoints = @('sql01:1433', 'lic01:27000', 'files01:445') }
    )
}

The script needs the Hyper-V module on the replica host, Windows Server 2016 or later guests (a PowerShell Direct requirement), a guest credential valid in the isolated environment, and replicas that have finished initial replication.

powershell
<#
.SYNOPSIS
    Runs a Hyper-V Replica test failover for a list of VMs and verifies services and dependencies inside each guest.
.DESCRIPTION
    Reads a plan file (.psd1) listing VMs in boot order with the services each must run and the host:port
    endpoints each must reach. For every VM it checks replication health, runs Start-VMFailover -AsTest,
    starts the test VM, waits for PowerShell Direct to respond, then checks services with Get-Service and
    endpoints with Test-NetConnection from inside the guest. When all VMs are done it removes every test VM
    with Stop-VMFailover (unless -KeepTestVMs is set) and writes the results to a CSV file.
.PARAMETER PlanPath
    Path to the .psd1 plan file.
.PARAMETER GuestCredential
    Credential for the guests, valid inside the isolated test network.
.PARAMETER BootTimeoutMinutes
    How long to wait for each test VM to answer PowerShell Direct.
.PARAMETER SettleSeconds
    How long to wait after the guest answers before checking services, for delayed-start services.
.PARAMETER EvidencePath
    CSV file for the results.
.PARAMETER KeepTestVMs
    Leave the test VMs running for manual testing; clean up later with Stop-VMFailover.
.EXAMPLE
    .\Invoke-DRTestFailover.ps1 -PlanPath .\dr-plan.psd1 -GuestCredential (Get-Credential)
.NOTES
    Author  : Thomas Lasswell (https://www.techcolumnist.com)
    Version : 1.0 (2026-07-22)
    Requires: Hyper-V module on the Replica server, Windows Server 2016+ guests
#>

[CmdletBinding()]
param (
    [Parameter(Mandatory = $true)]
    [string]$PlanPath,

    [Parameter(Mandatory = $true)]
    [pscredential]$GuestCredential,

    [Parameter(Mandatory = $false)]
    [int]$BootTimeoutMinutes = 15,

    [Parameter(Mandatory = $false)]
    [int]$SettleSeconds = 120,

    [Parameter(Mandatory = $false)]
    [string]$EvidencePath = (Join-Path -Path (Get-Location).Path -ChildPath ('DRTest-{0:yyyyMMdd-HHmm}.csv' -f (Get-Date))),

    [switch]$KeepTestVMs
)

$ErrorActionPreference = 'Stop'
$plan = Import-PowerShellDataFile -Path $PlanPath
$results = [System.Collections.Generic.List[object]]::new()
$testedVMs = [System.Collections.Generic.List[string]]::new()
$runClock = [System.Diagnostics.Stopwatch]::StartNew()

try {
    foreach ($entry in $plan.VMs) {
        $row = [ordered]@{
            VM           = $entry.Name
            Replication  = 'Normal'
            TestVM       = ''
            ReadySeconds = $null
            FailedChecks = ''
            Result       = 'Pass'
        }

        # Replication health first: a Critical replica is a finding before any test runs.
        foreach ($health in 'Warning', 'Critical') {
            if (Get-VMReplication -VMName $entry.Name -ReplicationHealth $health -ErrorAction SilentlyContinue) {
                $row.Replication = $health
            }
        }

        if ($row.Replication -eq 'Critical') {
            $row.Result = 'Fail'
            $row.FailedChecks = 'Replication health is Critical; not tested'
            $results.Add([PSCustomObject]$row)
            continue
        }

        # Create the test VM; Hyper-V names it after the replica with ' - Test' appended.
        Start-VMFailover -VMName $entry.Name -AsTest -Confirm:$false
        $testVM = Get-VM -Name "$($entry.Name) - Test" -ErrorAction SilentlyContinue | Select-Object -First 1

        if (-not $testVM) {
            $row.Result = 'Fail'
            $row.FailedChecks = 'Test VM was not created'
            $results.Add([PSCustomObject]$row)
            continue
        }

        $testedVMs.Add($entry.Name)
        $row.TestVM = $testVM.Name
        $clock = [System.Diagnostics.Stopwatch]::StartNew()
        Start-VM -VM $testVM

        # Wait until the guest answers over PowerShell Direct.
        $ready = $false

        do {
            try {
                Invoke-Command -VMId $testVM.VMId -Credential $GuestCredential -ScriptBlock { $true } -ErrorAction Stop | Out-Null
                $ready = $true
            } catch {
                Start-Sleep -Seconds 15
            }
        } while (-not $ready -and $clock.Elapsed.TotalMinutes -lt $BootTimeoutMinutes)

        if (-not $ready) {
            $row.Result = 'Fail'
            $row.FailedChecks = "Guest did not answer PowerShell Direct within $BootTimeoutMinutes minutes"
            $results.Add([PSCustomObject]$row)
            continue
        }

        $row.ReadySeconds = [int]$clock.Elapsed.TotalSeconds
        Start-Sleep -Seconds $SettleSeconds

        # Check services and dependencies from inside the recovered guest.
        $checks = Invoke-Command -VMId $testVM.VMId -Credential $GuestCredential -ArgumentList $entry.Services, $entry.Endpoints -ScriptBlock {
            param ($services, $endpoints)

            foreach ($serviceName in $services) {
                $service = Get-Service -Name $serviceName -ErrorAction SilentlyContinue
                [PSCustomObject]@{ Check = "service $serviceName"; Passed = [bool]($service -and $service.Status -eq 'Running') }
            }

            foreach ($endpoint in $endpoints) {
                $target, $port = $endpoint -split ':'
                $reachable = Test-NetConnection -ComputerName $target -Port $port -InformationLevel Quiet -WarningAction SilentlyContinue
                [PSCustomObject]@{ Check = "tcp $endpoint"; Passed = [bool]$reachable }
            }
        }

        $failed = @($checks | Where-Object { -not $_.Passed })

        if ($failed.Count -gt 0) {
            $row.Result = 'Fail'
            $row.FailedChecks = ($failed.Check) -join '; '
        }

        $results.Add([PSCustomObject]$row)
    }
} finally {
    # Remove the test VMs only after every dependency has been exercised, or when the run stops on an error.
    if (-not $KeepTestVMs) {
        foreach ($name in $testedVMs) {
            Stop-VMFailover -VMName $name -Confirm:$false
        }
    }
}

$results | Export-Csv -Path $EvidencePath -NoTypeInformation
$results | Format-Table -AutoSize
Write-Output "Test run took $([int]$runClock.Elapsed.TotalMinutes) minutes. Evidence: $EvidencePath"

A first run against a plan like the one above tends to look something like this, and every Fail row is a finding for the report:

text
VM      Replication TestVM          ReadySeconds FailedChecks                      Result
--      ----------- ------          ------------ ------------                      ------
dc-dr01 Normal      dc-dr01 - Test           212                                   Pass
sql01   Warning     sql01 - Test             348                                   Pass
lic01   Normal      lic01 - Test             190 service <license-service-name>    Fail
app01   Normal      app01 - Test             241 tcp files01:445                   Fail

Stop-VMFailover on a test failover turns the test VM off and deletes it; it does not touch the replica or the primary. For Azure Site Recovery the equivalent pair is Start-AzRecoveryServicesAsrTestFailoverJob (against a recovery plan, with -AzureVMNetworkId pointing at the isolated network) and Start-AzRecoveryServicesAsrTestFailoverCleanupJob, and the in-guest checks can run the same way over whatever remote access you have to the test network.

The Runbook Checklist

The script covers the mechanical part. The rest is a checklist I keep with the DR plan and fill in during every test:

Before the test

  • Replication health is Normal for every VM in scope (Get-VMReplication, or Replicated items in the vault), and you know how far behind the last replication is.
  • Recovery point chosen deliberately: latest, or the latest application-consistent point for databases.
  • Isolated test network in place, using the production address range and with no route back to production. Test switch set on every replicated adapter.
  • DC and DNS first in the boot order; the DR DC is a global catalog and holds (or can seize) the FSMO roles the test needs.
  • Plan file reviewed against the application owners' list of dependencies, including licence servers, file shares and external APIs.
  • Guest credentials that work without production DCs are available, and the break-glass account has been tested.

During the test

  • Start time, each VM's ready time and each failed check recorded (the script's CSV).
  • Every manual intervention written down with who did it and why. Each one counts against recovery confidence, however quickly it was done.
  • Application owners sign in and run a real transaction, not just a ping.
  • Licence activation and certificate validation checked on every application that has either.

After the test

  • Test VMs removed (Stop-VMFailover, or Cleanup test failover in Site Recovery) and replication confirmed healthy again.
  • Each finding logged as a fix with an owner and a date, and the plan file updated with any dependency the test discovered.
  • A retest of each fixed item scheduled as soon as the fix is in, not at next year's test.

Running It Again, On Purpose

Every finding from a real test becomes a fix, and every fix needs to be verified by running the test again — not next year, but as soon as the fix is in, on the specific piece that broke. A test failover that costs no downtime and is driven by a script makes that cheap enough to actually do. A DR plan that's only ever been fully exercised once, however well it went, is still mostly a hypothesis about everything the last test didn't happen to touch.

References