When the CDN Goes Dark: Building an Image Pipeline That Survives Black Friday at 3 A.M.
Photo: engineer monitoring multiple dashboards in dark network operations center at night, via thumbs.dreamstime.com
At 3:04 a.m. on the Saturday after Thanksgiving, your phone buzzes. The status page for your image CDN is a wall of yellow. Four minutes later the app's category carousels are rendering as gray boxes, and by 3:20 the support queue has its first screenshot from a customer who just wanted to know if the boots were in stock.
Nobody pages you when a font fails to load. Everybody pages you when the product photo doesn't.
That asymmetry is the whole argument for treating image delivery like a database instead of a nice-to-have. You would never run a checkout flow against a single Postgres instance with no replica and no backup. Yet plenty of US retailers and SaaS companies route every single visual asset through one CDN account, one origin bucket, and one engineer who knows how the whole thing is wired.
Why images deserve a seat at the disaster-recovery table
A modern storefront or app screen is mostly pixels. On a typical product detail page, images account for 60 to 75 percent of transferred bytes. When the image layer fails, the page technically loads: the HTML is fine, the CSS is fine, the buttons work. It just looks broken. And to a shopper comparing two tabs on their phone, "looks broken" and "is broken" are the same thing.
The stakes get sharper around the calendar. Black Friday lands on November 27 this year and Cyber Monday on November 30, and traffic on those days is concentrated into hours where every percentage point of conversion is worth real money. An image outage during that window isn't an uptime metric. It's a revenue event.
The four layers of a failover plan that actually fails over
1. DNS-level steering between providers
Multi-CDN only works if traffic can move without a human. That means a DNS layer (or a traffic manager) that health-checks your primary image host and shifts CNAMEs to a secondary provider when error rates or latency breach a threshold. Keep TTLs low enough that a cutover takes minutes, not hours, and make sure your secondary provider already has your certificates and configuration. A standby CDN you've never sent traffic to is not a standby. It's a rumor.
2. Origin fallback that doesn't fall over
If your origin is the same region as your CDN's edge nodes, an outage upstream takes both out. Keep at least one replica in a different region, replicate your transformation cache, and confirm that your image URLs still resolve when the transformation service is unreachable (serve the original, just larger). A degraded-but-image-present experience beats a fast 404 every time.
3. Cache warming before the traffic, not after
Failover to a cold cache is its own outage. The secondary provider has none of your assets, so the first wave of requests stampedes your origin. Warm the top assets by traffic before peak windows: your top 500 product images, category headers, hero banners, and any campaign creative that's about to go live. A simple script that walks your sitemap or analytics export and requests each URL through the alternate CDN will do more for your Black Friday than most performance budgets.
4. Health checks that test the right thing
A ping to the CDN's root domain tells you almost nothing. Check a real asset path from multiple regions, verify the response body is an actual image and not an HTML error page with a 200 status, and alert on time-to-first-byte for image endpoints rather than raw availability. Image CDNs fail in weird, partial ways: one POP degrades, one transformation endpoint breaks, one region loses cache. Your checks should be able to tell those apart.
The first fifteen minutes, written down in advance
The runbook matters more than the architecture, because the architecture only helps if someone remembers it exists at 3 a.m.
- Confirm scope. Is it every image, one format, one region, or one transformation rule? Pull a handful of representative URLs and check status codes and content types.
- Declare. Say the word "incident" out loud in the channel. Assign one person to customer communication and one to technical mitigation. The same person doing both does neither well.
- Flip the switch. Move DNS to the secondary provider or enable the failover rule your traffic manager already has configured. Do not start editing configs you haven't touched in eight months.
- Protect the origin. Turn on aggressive edge caching, serve originals instead of transformed variants, and consider temporarily dropping the least-critical image sizes so the origin isn't hammered by ten breakpoints per asset.
- Degrade gracefully. Swap hero carousels for static placeholders only as a last resort, and make sure your placeholders are lightweight and accessible rather than a 2 MB "image unavailable" graphic.
- Watch the second-order effects. Checkout, email templates, and paid social all pull from the same library. An outage that only breaks the PDP is lucky. Assume it's broader.
Rehearse it, or lose it
Tabletop exercises feel like theater until the first real outage, at which point they're the only reason anyone knows who calls the CDN account rep. Run a quarterly drill: move a small percentage of production image traffic to the secondary provider during business hours, measure how long the cutover takes, and time how long it takes the on-call engineer to find the runbook. Then fix the thing that made you wince.
Also decide, in advance, what you're willing to spend on failover. Running two CDNs at low traffic volume mostly costs configuration effort. Running two at full volume costs real money, which is why many teams keep the secondary at a partial traffic share and accept a slower cutover in exchange for a smaller bill.
After the all-clear
Write it down while it's fresh: what failed, what you assumed, what actually happened, and which of your dashboards told you the truth. Image pipelines are usually assembled incrementally, one integration at a time, and the weak link is almost never the part you thought about most recently.
The test isn't whether your CDN goes down. It will. The test is whether the lights in your storefront stay on when it does.