Back to BlogRun Repeatable WAN Failover Tests With SLA Metrics and Automation

Run Repeatable WAN Failover Tests With SLA Metrics and Automation

NTNetverge TeamNetverge editorial teamUpdated
WAN redundancy testingbackup connection testingWAN reliability assessmentdisaster recovery testingfailover simulation

A robust WAN failover test uses a mix of forwarding-plane probes such as BFD or routed probes plus at least one application-level health check, and it passes when time-to-failover and application continuity meet your SLA thresholds. Run simulated cutovers monthly and reserve live, unannounced cutovers for quarterly or annual exercises. Anything less leaves you guessing whether your redundant circuit actually works when it matters.

TL;DR:

  • Combining forwarding-plane probes like BFD with application-level checks provides the most reliable detection of WAN failures by catching both path and session issues.
  • Conducting regular simulated and live cutovers, with pre-defined thresholds and timestamps, ensures accurate measurement of detection, switchover, and application recovery times.
  • Automating probes and trigger commands across multiple sites reduces error, increases consistency, and creates audit-ready logs for SLA validation.
  • Testing should account for application types, with real-time applications requiring tighter thresholds and careful failback procedures to prevent session disruptions.
  • Verify diverse physical paths and avoid asymmetric routing to prevent false positives, false negatives, and simultaneous circuit failures that undermine redundancy.

Table of Contents

Connection Tests and Probe Types: What Each One Proves

Every probe type answers a different question, and none of them answers all of them. Picking the wrong one gives you false confidence.

  • ICMP/ping confirms basic reachability but says nothing about application health and can be filtered or rate-limited by upstream providers.
  • ARP checks verify local gateway reachability only, so they miss failures further upstream.
  • DNS probes confirm resolution is working but can succeed even when the actual application path is broken.
  • HTTP/transactional checks are the closest proxy for real user experience, since they test the full path including NAT and firewall state.
  • BFD operates as an OAM mechanism built for rapid, sub-second failure detection on the forwarding plane, according to RFC 5880, but it was not designed as an application liveness check across the public Internet without special handling for encapsulation and authentication, per RFC 5883.

For most enterprise WANs and MSP-managed sites, pair BFD or a rapid routed probe with at least one application-level check. That combination catches both forwarding-plane failures and the session-breaking side effects that BFD alone will not see.

Pre-Test Checklist: Targets, Routing, and Rollback

Bad test design produces bad data, or worse, an unplanned outage. Before you touch a live circuit, lock down these basics.

  1. Select at least three monitoring targets: a public DNS resolver, an upstream NATed endpoint, and an internal application server, so no single point of failure skews your results.
  2. Confirm how routing, NAT, and firewall state tables behave during a path change, since stateful devices can silently drop sessions that a simple ping would call healthy.
  3. Schedule a maintenance window or isolate the test to a lab segment, and document who to call and what command reverses the change if something goes wrong.
  4. Configure monitoring to capture both control-plane and data-plane timestamps so you can separate detection delay from actual traffic loss.

Pro Tip: Keep a rollback command pre-typed and ready in a second terminal window before you initiate any live cutover.

Step-by-Step Failover Test Procedures

Consistent test runs produce comparable data. Run these in order, and log every timestamp as you go.

Controlled cut test

  1. Disable the primary uplink at the interface level, not by unplugging a cable, so the disable event is logged with a timestamp.
  2. Watch for the monitoring system to flag the primary as down, and note the exact detection timestamp.
  3. Confirm traffic shifts to the backup path by checking active sessions on the firewall or router, not just the routing table.
  4. Run a continuous ping or transactional probe through the cutover window to measure the loss burst.

Simulated fail test

Use administrative interface shutdown commands or a route withdrawal when you want a repeatable, low-risk test that does not require physically touching hardware. This method suits monthly test cadences where a full circuit pull is unnecessary.

BFD session disruption test

Trigger a BFD session drop by disabling the interface running the session, then capture the time between the last received control packet and the moment the routing protocol reacts. Because BFD common intervals are negotiated between peers and can fall back to a slower shared timer when endpoints differ, per RFC 7419, confirm your actual negotiated interval before trusting claims of very fast detection.

Application continuity checks

For TCP-based applications, monitor whether existing sessions survive the cutover or require a client reconnect. For SIP, watch for call drops or re-registration delays. For VPN tunnels, confirm the tunnel renegotiates without requiring a manual restart on the client side.

Data to collect

Record the primary-down timestamp, the detection timestamp from your monitoring system, the traffic-shift timestamp from session tables, and the first successful transaction on the backup path. Together these let you calculate time-to-detect and time-to-forward as separate, defensible numbers instead of one blended guess.

Metrics and Acceptance Criteria: Pass or Fail

A test without thresholds is just an observation. Define your numbers before you run the test, not after you see the results.

  • Detection time: how long between the failure event and your monitoring system flagging it.
  • Switchover time: how long between detection and traffic actually flowing over the backup path.
  • Packet-loss burst: the volume of dropped packets during the transition window.
  • Post-failover stabilization time: how long until latency and jitter return to baseline on the backup circuit.
  • Application reconnect time: how long a session takes to resume normal operation for the end user.

A common illustrative threshold set: detection and switchover combined at or under your configured RTO, a packet-loss burst capped at a level your critical applications tolerate, and application reconnect within an acceptable session window. Treat these as a starting framework you adjust to your own SLA, not a fixed industry number.

Synchronized timestamps, ideally from NTP-disciplined sources, are what make these metrics defensible rather than approximate. Feed the results back into your WAN performance monitoring baselines and use them to update your documented RTO and RPO targets, a process the RPO and RTO guidance from Great Plains Networking breaks down in more detail for teams sizing recovery targets against budget.

Common Failure Modes and Troubleshooting

When a test comes back with bad numbers, the cause usually falls into one of a few familiar buckets.

  • False positives from ICMP filtering: some upstream providers rate-limit or block ping entirely, making a healthy path look dead.
  • Shared physical paths: two "redundant" circuits that ride the same conduit, pole, or carrier hand-off will fail together; verify diverse physical routing before trusting the redundancy on paper.
  • Asymmetric routing: traffic that goes out one path and returns on another can trip stateful firewalls into dropping sessions that look fine at the routing layer.
  • BFD fails but applications succeed: this usually points to a forwarding-plane blip that recovered before the application layer noticed, worth investigating but not always worth an alert escalation.

Pro Tip: When BFD and application probes disagree, check the interface error counters first. A flapping link often explains the mismatch before you need to dig into routing tables.

Operationalizing Failover Tests

One-off tests tell you almost nothing about drift over time. Build a cadence and a runbook instead.

  1. Run simulated failover tests on a regular, frequent cadence, scheduling a full live cutover test periodically, and deeper exercises following NIST SP 800-34 contingency planning guidance on testing, training, and exercises.
  2. Build a runbook skeleton with five sections: pre-checks, test actions, verification steps, rollback procedure, and a postmortem with named owners, similar in structure to the approach outlined in a network incident response guide.
  3. Automate what you can: scheduled probes, automatic ticket creation on threshold breach, and a results dashboard that shows trend lines across test cycles.

Vergepoints and scheduled probes within Netverge map directly onto steps one and three, generating timestamped evidence and triage tickets without a technician manually running each cycle.

Failback Behavior and Sequencing

Failover gets most of the attention, but failback is where session-breaking mistakes happen. When the primary circuit returns, do not assume traffic should snap back immediately.

Configure a stabilization delay before failback triggers, typically a few minutes of confirmed primary uplink health, so a flapping link does not bounce traffic back and forth repeatedly. Flapping between paths is worse for application stability than staying on the backup circuit a little longer than necessary.

Sequence failback carefully for stateful services. TCP sessions and VPN tunnels that survived the original failover may not survive an abrupt failback if the routing change happens faster than the session state can migrate. Consider a staged failback where lower-priority traffic classes move first, followed by critical applications once you confirm the primary path is stable under load, not just reachable.

Log the failback event with the same rigor as the failover event: timestamp the primary-restored signal, the failback trigger, and the first successful transaction back on the primary path. This gives you a complete before-and-after picture instead of only measuring half the cycle.

Test failback deliberately as part of your regular cycle, not as an afterthought. A circuit that fails over cleanly but fails back badly will show up in your metrics only if you are actually measuring the return trip.

Failback Behavior and Sequencing — overview diagram

Automated WAN Failover Testing Tools and Scripting Techniques

Manual failover tests do not scale past a handful of sites, and they invite human error into a process that should be repeatable. Scripting the test cycle removes both problems.

A basic automation approach uses a scheduler to trigger interface shutdown commands via API or CLI automation on a defined cadence, paired with a monitoring system that logs the resulting detection and recovery timestamps automatically. Many router and firewall platforms expose an API for this exact purpose, letting you script the primary-down event, wait for a defined observation window, then script the rollback.

Combine that trigger with continuous probes running throughout the test window: a scripted ping sweep, a scheduled DNS query, and a transactional HTTP check hitting a real application endpoint. Correlating the three probe types in one script gives you a single test report instead of three disconnected logs you have to reconcile by hand.

For teams managing many sites, a platform-based approach replaces custom scripts with scheduled test templates that run automatically across a whole client base. This is where dashboards and automated ticket generation matter most, since the goal is not just running the test but making sure a failed threshold actually reaches someone who can act on it, without an engineer manually reviewing every log file after every cycle.

Impact Analysis of Failover on Traffic and Applications

Not every application experiences failover the same way, and treating them identically in your test plan will miss real problems.

Bulk data transfer and file syncing tend to tolerate a failover cleanly, since TCP retransmission absorbs a brief interruption without the end user noticing. Real-time traffic is far less forgiving. Voice and video calls over SIP can drop entirely during a switchover window that a file transfer would shrug off, so any WAN redundancy testing plan covering voice traffic needs its own pass/fail threshold, tighter than what you'd apply to background traffic.

VPN tunnels sit in the middle. A site-to-site tunnel usually renegotiates automatically after a failover, but the renegotiation itself takes time, and during that window any application riding the tunnel sees a full outage, not a degraded connection. Test VPN failover separately from raw connectivity failover, since a "successful" circuit switchover can still mean several seconds of total application downtime while the tunnel rebuilds.

Database replication and other stateful backend traffic deserve particular attention, since a dropped session mid-transaction can leave data in an inconsistent state depending on how the application handles retries. If your environment runs anything transactional over the WAN, include an explicit test of that specific traffic type rather than assuming a general pass on ping and HTTP checks covers it.

Impact Analysis of Failover on Traffic and Applications — overview diagram

Logging and Alerting Best Practices During Tests

Good logging turns a failover test from a one-time observation into a comparable data point you can track over months.

Capture control-plane events (BFD state changes, routing table updates) and data-plane events (packet loss, latency spikes) as separate log streams, since blending them makes it harder to isolate whether a delay came from detection or from the actual traffic path. Timestamp everything against a common time source, ideally NTP-disciplined, so cross-referencing logs from different devices produces a coherent timeline instead of a set of inconsistent clocks.

During the test itself, suppress non-essential alerts tied to the expected primary-down event so your team is not paged for a condition you deliberately created. At the same time, keep alerting active for anything outside the expected scope, such as a secondary circuit also showing instability, since that is exactly the kind of hidden dependency a test is meant to expose.

After the test, archive the full log set alongside your metrics summary rather than just the pass/fail outcome. A future test that shows a slower detection time is only useful if you can pull the prior run's logs and compare the actual sequence of events, not just the final number.

Safety Precautions and Rollback Plans

Every failover test carries some risk of an unplanned outage, and the safeguards you put in place before the test matter more than anything you do during it.

Run your first tests in a maintenance window with stakeholders notified, even if you plan to move to unannounced live tests later once your team trusts the process. Confirm a rollback command is ready before you trigger the failure, not written down somewhere you'd need to search for under pressure.

Set a hard time limit for the test and a clear rule for who calls it off if something looks wrong early. A five-minute planned test that turns into a thirty-minute troubleshooting session because nobody had authority to abort it defeats the purpose of a controlled exercise. Guidance from CISA's NECP Spotlight on redundancy reinforces that documented backup procedures and rehearsed coordination are what actually prevent extended outages during real incidents, not just the existence of a redundant circuit.

Finally, test in a way that lets you isolate blast radius. Start with low-traffic hours or a subset of sites before extending the same test to a full production cutover across every location.

Choosing the Right Depth of Testing

Tighter BFD timers catch failures faster, but every millisecond you shave off detection adds noise from transient link blips that recover on their own. Reserve sub-second BFD tuning for circuits carrying voice or other real-time traffic, and lean on periodic application checks for everything else. MSPs managing many clients should tier test frequency by contract criticality rather than applying one aggressive standard everywhere, since the operational cost of chasing every false alarm outweighs the benefit for a low-priority site.

— Jim

How Netverge Helps You Scale Failover Testing

Running these tests manually across dozens of sites is where most teams fall behind, not on knowing what to test but on finding time to do it consistently. Netverge automates the parts of failover testing that eat the most technician time.

Netverge

  • Scheduled probes across BFD, DNS, and HTTP checks run automatically on your defined cadence, without a technician kicking off each cycle.
  • Hardware Vergepoints provide on-site visibility at each location, so you get local confirmation of failover behavior instead of relying only on remote checks.
  • Automated ticket generation and dashboards turn every test cycle into timestamped evidence you can hand to a client or auditor without rebuilding the report by hand.

That means fewer manual test runs, consistent documentation for SLA conversations, and faster postmortems when a real outage happens. Netverge's Starter Package is available starting at $299 per month, with Hardware Vergepoints priced separately on the pricing guide. Check current availability and get started at Netverge.

Sources

For deeper technical grounding, review NIST SP 800-34 on contingency testing, RFC 5880 on BFD operation, and CISA's redundancy guidance referenced above.

FAQ

What Is WAN Failover?

WAN failover is the automatic switch from a primary internet or WAN circuit to a backup connection when the primary path becomes unavailable. The switch relies on connection monitoring, typically a combination of routed probes and application checks, to detect the outage and redirect traffic.

How Do I Perform a Failover Test?

Disable the primary uplink at the interface level in a controlled maintenance window, then confirm your monitoring system detects the outage and traffic shifts to the backup circuit. Capture timestamps for detection, switchover, and application recovery so you can measure the results against your acceptance criteria.

What Is a WAN SLA?

A WAN SLA is the service level agreement that defines the performance and availability commitments for a WAN connection, often including uptime targets and, for redundant setups, an expected time-to-failover. Test results should be measured directly against these thresholds to confirm the circuit meets its contracted commitments.

Can UniFi Gateways Perform WAN Failover?

Many gateway platforms, including UniFi devices, support WAN failover configurations that monitor a primary connection and switch to a secondary one when it fails. The specific detection methods and timers vary by platform and firmware version, so check your device's current documentation for supported probe types and configuration options.

Recommended