Playbook: Responding to Cloudflare Outages (SRE Runbook)

Operational playbook for SREs to fail over, secure, and communicate during CDN/provider outages like the Jan 2026 Cloudflare incident.

Hook: When a single CDN outage can break your entire platform, what’s your plan?

If your users can’t reach critical features because a CDN or provider is down — and that outage cascades across authentication, APIs, and static assets — your SRE and security teams need an actionable, testable playbook. Recent outages in late 2025 and the January 2026 Cloudflare-related incident that disrupted major platforms demonstrated how quickly a single third-party failure can become an organizational emergency. This playbook gives SREs and security teams a step-by-step operational guide to fail over, protect users, and communicate effectively when Cloudflare-dependent services (or any major CDN/provider) go down.

Executive summary: What to do first (the inverted pyramid)

Detect and confirm — quickly determine scope and root cause.
Activate your incident response (IR) playbook — roles, runbooks, and communications.
Fail over traffic and critical services using pre-tested DNS and routing strategies.
Protect users and data — authentication fallback, rate limits, and WAF adjustments.
Communicate clearly — internal, customer-facing, and upstream provider engagement.
Post-incident: metrics, RCA, and resilience investments (multi-CDN, multi-DNS, backups).

1. Rapid detection & scope verification

Time-to-detect determines downtime impact. Use multiple signals to confirm a provider outage:

External monitoring: Synthetic checks from multiple regions (HTTP, TLS, TCP). If several regions see failures simultaneously, suspect CDN/edge provider impact.
Internal telemetry: Error rates, 502/503 spikes, origin logs showing no incoming edge requests, and WAF logs. Compare present traffic with expected baseline.
Third-party feeds and social signals: Vendor status pages, Twitter/X, Mastodon, and reputable news outlets. In Jan 2026, public reporting and vendor status pages confirmed a widespread Cloudflare disruption for many services.
Direct vendor contact: Open a high-priority support channel (phone/Slack/portal) immediately; don’t rely solely on the public status page.

Quick commands to validate scope

Check DNS resolution and whether the CDN is responding: dig +short example.com A
Confirm HTTP response headers to see if traffic is hitting the edge: curl -I https://example.com
Test from multiple vantage points: curl --resolve example.com:443:1.2.3.4 https://example.com

2. Incident activation & roles

Immediately trigger your Incident Response (IR) process. Assign clear roles and time-boxed tasks:

Incident Lead: Own triage, communications cadence, and business decisions.
SRE Lead: Execute failover runbooks, traffic routing, DNS changes, and coordination with vendor NOC.
Security Lead: Evaluate WAF, DDoS protections, authentication fallback, and risk of log loss or data exposure.
Comms Lead: Produce internal and external messages; coordinate legal and customer-success where needed.
On-call engineers: Rapidly implement technical mitigations and validate results.

3. Failover strategies: DNS & traffic routing

Failover must be planned and rehearsed. These are the practical options, ordered by reliability and complexity.

Option A — Multi-DNS + low TTL (fastest for simple cutovers)

Maintain a secondary DNS provider with preconfigured records and low TTLs for critical records (e.g., 60s) so you can switch A/AAAA or CNAME targets quickly.

Use a DNS provider that supports programmatic API changes and health checks (AWS Route 53, NS1, Cloud DNS).
Pre-provision records pointing to an alternate CDN, load balancer, or origin. Example: primary CNAME to cloudflare-cdn.example -> secondary CNAME to fastly-cdn.example.
TTL guidance: set critical records to 60–300s in production only where your change process and cache behavior accept short TTLs. For broader scale, use 300s to reduce churn.
Note: DNS propagation and resolver caching still add variability; test across major resolvers (Google DNS, Cloudflare 1.1.1.1, ISP resolvers).

Option B — Multi-CDN with intelligent steering

Deploy multi-CDN with a global traffic manager that can steer traffic based on provider health, latency, and cost.

Benefits: reduced single-vendor blast radius, performance optimization, and provider-level outages containment.
Implementation tips: keep a single canonical origin, use consistent TLS certificates (or edge TLS across providers), and ensure cache key compatibility to minimize cache misses during failover.
Test monthly: simulate failovers and measure cache warming times.

Option C — Origin direct & static asset fallback

If the edge is down, route critical traffic directly to origin or an alternate object store (S3/Blob) with pre-signed URLs or a short-lived auth proxy.

Pre-warm origin capacity: ensure autoscaling policies and connection limits are tested.
Serve static pages from object storage static-hosting or a minimal origin cluster with reduced functionality but acceptable UX (read-only mode for user timelines, for example).
Use HTTP response headers to minimize origin caching issues: set Cache-Control appropriately and use a CDN-agnostic cache-key scheme.

Option D — Anycast & BGP routing (advanced)

For large-scale, multi-cloud infrastructures: use BGP routing and anycast announcements to shift traffic between providers quickly. This requires network engineering expertise and pre-established peering.

4. Security & user protection during failover

Outages increase attack surface. Prioritize protecting user data and authentication flows.

Authentication fallback: If SSO or OAuth depends on the CDN edge, activate a fallback auth endpoint routed via alternate DNS. Keep refresh-token lifetimes long enough to avoid mass forced re-logins.
WAF and rate limits: Avoid disabling WAF globally. Instead, apply targeted relaxations (example: reduce strict bot checks that block legitimate health checks) and increase rate limits only for trusted IP ranges.
Session integrity: Monitor for session anomalies and enable additional logging for suspicious activity. If logs are delivered through the CDN, ensure a parallel logging pipeline from origin to your SIEM.
Data writes: If primary storage paths are compromised, pause non-essential write operations and queue them in durable message queues (Kafka, SQS) until integrity checks pass.

5. Communications: internal, external, and legal

Clear communication reduces churn and support load. Use templates and a cadence model.

Internal

Post updates every 15–30 minutes during active triage. Include scope, impacted services, mitigation in progress, and next steps.
Share observable metrics and expected ETA for next update. Keep messages short and factual.

External (customers and users)

Publish a short, clear status update on your status page and social channels every 30–60 minutes. Use a template: what happened, who’s impacted, what we’re doing, and expected next update time.
If you have an incident timeline, store it on your status page and update it as you learn more.
For enterprise customers, send targeted emails with specific mitigation steps and support contact info.

Regulatory & Legal

Assess data breach/regulatory notification requirements early. If the CDN outage could affect availability SLAs or contract obligations, involve legal and compliance teams.
Document all decisions and evidence — logs, timestamps, and vendor communications for after-action reviews.

6. Tactical runbook: step-by-step checklist

Below is a concise, ordered runbook you can follow during the first 90 minutes of an outage.

Detect — Confirm outage via synthetic checks and internal telemetry.
Activate — Trigger IR, assign roles, and open a conference bridge (video + persistent chat).
- Notify leadership and customer success.
Confirm vendor impact — check vendor status + open expedited support ticket.
- Record ticket ID and SLA escalation path.
Decide failover scope — full cutover vs. partial (APIs vs. static assets).
- Choose minimal viable functionality to restore quickly.
Execute DNS/Traffic changes — using pre-tested scripts with dry-run capability.
- Change TTL if needed, then switch A/CNAME to secondary target.
- Validate global reachability from major regions.
Protect auth and data — apply temporary rate limit adjustments and queue writes.
Communicate externally — status page update + social post.
Monitor — confirm traffic stabilization, error rate reductions, and customer-reported improvements.

7. Testing & rehearsal: make failovers reliable

Playbooks only work if practiced. Adopt a cadence of scheduled tests and unannounced drills.

Monthly smoke tests: Validate DNS failovers, health checks, and origin direct routing.
Quarterly chaos engineering: Simulate provider outages using controlled chaos tools to measure recovery time and cache-warm metrics.
Post-test review: Update runbooks with observed edge cases and refine TTLs and automation scripts.

8. Architecture and backup recommendations (Cloud backup pillar)

Long-term resilience requires architecture changes and solid backup strategies.

Multi-CDN + Multi-DNS

Combine two reputable CDNs and at least two authoritative DNS providers. Ensure programmatic control and consistent SSL/TLS across providers.

Origin hardening and backups

Use immutable object versioning in object stores (S3 Object Lock, GCP Object Versioning) and store backups in a different cloud region or provider.
Maintain warm backup origins with automated sync and failover automation that can be exercised via CI pipelines.

Edge compute & auth design

Design auth flows that can route to alternate endpoints if the edge provider fails. Avoid putting critical auth token issuance exclusively behind a single provider’s edge functions.

Immutable & encrypted backups

Keep at least three copies of critical data across multiple providers and regions. Use KMS-encrypted backups with stored key rotation policies.

9. Third-party risk: continuous inventory and contracts

Outages highlight supply-chain risk. Maintain a live third-party inventory:

Tag criticality for each vendor (A/B/C) and define backup plans per vendor.
Negotiate SLAs and incident escalation procedures; expect some vendors to provide guaranteed credits for major outages.
Run annual PEN tests and dependency audits to discover hidden chains (e.g., CDN-integrated auth flows, edge logging).

10. Post-incident actions and KPIs

After containment, conduct a structured RCA and update systems based on learnings:

Collect logs, vendor communication artifacts, and metrics (MTTD, MTTR).
Run a blameless postmortem within 72 hours. Publish a summary to stakeholders and customers that includes technical causes and concrete mitigations.
KPIs to track: mean time to detect (MTTD), mean time to failover (MTTFo), and user-facing downtime minutes. Track cost/benefit of multi-provider setups.

Real-world example: lessons from Jan 2026

In January 2026, a Cloudflare-related disruption affected multiple high-profile services. The event underscored several realities:

When CDNs provide more than caching (edge auth, WAF, logging), outages have larger blast radii.
Public reporting and third-party detection helped companies validate vendor impact quickly.
Organizations with multi-DNS/CDN strategies and origin-direct fallbacks experienced significantly lower user impact.

"Outages in late 2025 and early 2026 show the importance of designing for provider failure modes — not just provider performance."

2026 trends to prepare for (short-term predictions)

Consolidation risks: More platform bundling by CDN providers increases systemic risk — diversify where it matters.
Edge as a service: Edge compute adoption will grow; ensure critical logic can be rerouted off-edge quickly.
Regulatory scrutiny: Expect more requirements for incident reporting and third-party risk management.
Automated failover tooling: Increased emergence of vendor-agnostic steering platforms and DNS automation tools — invest in test-driven automation now.

Playbook cheat-sheet (printable)

Detect: multi-region synthetic checks ✅
Activate: IR call + roles in 5 min ✅
Failover: DNS switch / multi-CDN redirect ✅
Protect: auth fallback + queued writes ✅
Communicate: status page + customer emails ✅
Postmortem: 72-hour blameless RCA ✅

Actionable takeaways

Build and practice a DNS-based failover that can be executed in under 15 minutes.
Maintain at least two DNS providers and one alternate CDN or origin path for critical services.
Design auth flows that can be decoupled from a single edge provider under stress.
Invest in synthetic monitoring from diverse regions and automate vendor health checks into your alerting rules.
Run monthly failover drills and quarterly chaos experiments to validate the entire chain from DNS to user experience.

Closing: make provider outages a known, exercised failure mode

CDN and provider outages will continue to occur in 2026. The difference between a long, reputation-damaging outage and a short, contained incident is preparation. Use this playbook to codify your failovers into automated, tested runbooks. Focus on short TTL DNS strategies, multi-DNS/multi-CDN architecture, origin hardening, and clear communication. Keep drills frequent and postmortems blameless.

Call to action: If your team doesn’t have an executable, tested DNS failover and origin fallback plan today, schedule a 90-minute tabletop drill this week. Start by mapping your critical CDN-dependent flows, auditing DNS TTLs, and pre-authorizing a secondary DNS/CDN cutover script. Contact support or request a runbook template to get a tested playbook you can run during your next provider outage.

Playbook: What to Do When Cloudflare-Dependent Services Like X Go Down

Hook: When a single CDN outage can break your entire platform, what’s your plan?

Executive summary: What to do first (the inverted pyramid)