Six hours of downtime. One Azure region. Zero redundancy. I got the call at 11 PM on a Tuesday and spent the rest of the night explaining to a very unhappy client why their entire workload had gone offline – because their infrastructure team had treated Azure like it was a single physical datacenter.This is that post-mortem. And honestly, it’s one I should have forced them to read before they went anywhere near a cloud subscription.
What Went Down
The client runs a mid-sized e-commerce platform. They’d migrated from on-prem to Azure roughly eight months prior. The migration went “fine” – everything spun up, apps connected, team celebrated. But nobody stopped to ask whether the architecture actually made sense.
At around 9:45 PM, Azure’s East US region experienced a partial service disruption affecting Compute resources. Standard stuff – it happens. The problem was that every single VM, every database, every load balancer – all of it – lived in one availability zone within that region. No secondary region. No availability sets. No zone redundancy. When East US coughed, they flatlined.
The business impact was immediate. The e-commerce site went dark during one of their promotional windows. By the time we restored full service, they’d logged an estimated six-figure revenue loss and burned significant goodwill with their customer base.
The Timeline
Here’s how the night unfolded.
21:45 – Azure begins experiencing degraded Compute performance in East US. No alerts fire on the client side because monitoring thresholds weren’t configured correctly.
22:03 – Their on-call engineer notices the site returning 503 errors. They restart a few VMs. Nothing changes.
22:17 – I get the call. They’ve already spent 15 minutes rebooting things and checking application logs. Nobody has checked the Azure Service Health dashboard yet.
22:25 – We pull up Azure Service Health. Active incident in East US. At this point, we know the problem is infrastructure-level, not application-level. That’s the bad news – there’s nothing we can do except wait or failover.
22:40 – We start assessing failover options. There are none pre-built. We begin manually standing up resources in East US 2 from scratch, working off ARM templates that thankfully existed.
01:12 – Full service restored in East US 2. Six hours from incident start to resolution. Far too long.
Root Cause: Azure Architecture Nobody Explained
The short answer is that nobody on the migration team actually understood how Azure structures its infrastructure. They knew how to click buttons in the portal. That’s not the same thing.
But let me be specific about the gaps, because each one is its own failure mode.
Regions, Geographies, and Why They’re Not Interchangeable
Azure organizes its global infrastructure into geographies – typically aligned to country or regulatory boundaries – and within those, individual regions like East US, West Europe, or Southeast Asia. Each region is a cluster of datacenters. When a region has problems, everything in that region can be affected.
The client picked East US because it was close to their users and checked out fine during testing. Reasonable. But they never considered what would happen if East US had an issue. The answer should have been “we fail over to East US 2 or Central US automatically.” Instead, the answer was “we sit and wait.”
This is the foundational mistake. Azure’s global distribution only helps you if you actually use more than one region for critical workloads. If you’re running everything in one spot, you’ve just built a slightly more expensive on-prem setup with worse physical access.
Availability Zones vs. Availability Sets
Even within a single region, Azure gives you tools to protect against localized failures. Availability Zones are physically separate datacenter facilities within the same region – independent power, cooling, and networking. Spreading your boxes across two or three zones means a single datacenter issue won’t take everything down at once.
Availability Sets are an older mechanism that distributes VMs across separate fault domains and update domains within a single datacenter. Less protection than zones, but still better than dumping everything onto the same physical rack. (Microsoft actually recommends zones over sets for new deployments – sets exist mostly for legacy compatibility.)
The client was using neither. Every box sat in the same zone, same datacenter, same everything. Microsoft’s SLA for a single VM – even with premium storage – assumes some level of hardware redundancy on their end, but it doesn’t protect you from regional disruptions. That’s your job.
For teams still working through where cloud fits into their responsibility model, the XaaS Cloud Service Models: Security Guide for IT Teams breaks down the IaaS shared responsibility boundary in useful detail.
Resource Groups and Subscriptions: Not Just Organizational Folders
The client had one resource group. Everything in it. No separation between production, staging, and dev. This wasn’t the cause of the outage, but it made recovery significantly harder.
Resource groups in Azure are the unit of management – you apply policies, permissions, and locks at the resource group level. You can also nuke an entire resource group in one operation, which is useful for cleanup and catastrophic when someone targets the wrong group. Having prod and dev in the same group is asking for a bad day.
They also ran a single subscription for everything. One billing scope, one set of service limits, one quota ceiling for the entire organization. Not ideal when you’re trying to isolate environments or track costs by department.
The lack of off-site data protection made things worse – their storage accounts had no geo-redundant replication configured, and there was no external backup of critical data. A regional outage that included data loss would have been catastrophic, not just inconvenient.
The Fix
We rebuilt the architecture over the following two weeks. Not a full migration – a restructuring. Here’s what changed.
Multi-Region Architecture With Defined Failover
We moved the production workload to an active-passive setup spanning East US and East US 2. The active region handles all traffic under normal conditions. The passive region maintains warm standby VMs and replicated storage. Azure Traffic Manager handles DNS-level failover if the primary region goes unhealthy.
This isn’t active-active – that adds complexity and cost that wasn’t justified for this client. Active-passive with automated failover delivers most of the resilience at a fraction of the operational overhead. This works well for workloads with moderate recovery time objectives, but not for anything where even five minutes of downtime is unacceptable.
Availability Zones for Core Compute
Within East US, we spread production VMs across zones 1, 2, and 3. The load balancer moved to a Standard SKU with zone-redundant configuration – they’d been running the Basic SKU, which doesn’t support zone redundancy at all. (Another oversight from the original migration that nobody caught until it mattered.)
The database tier moved to Azure SQL with zone-redundant backup enabled and geo-replication to the secondary region. Single-zone failures now have no user-visible impact.
Resource Groups Restructured
We split everything into separate groups: prod-rg, staging-rg, dev-rg, and infra-rg for shared networking components. Resource locks went on prod-rg – no delete operations without explicit approval. Mandatory tags enforced via policy so cost reporting actually reflected reality instead of one giant undifferentiated bill.
Part of the rebuild also involved reconsidering what needed to live in Azure at all. Some internal tooling moved to a dedicated NVMe-backed virtual server to cut cloud spend on low-traffic workloads that got no benefit from elastic scaling.
Monitoring That Actually Fires
We configured Azure Monitor with actionable alert thresholds – not just “VM is off” but “response time exceeds 2 seconds for 5 consecutive minutes” and “failed request rate exceeds 1% over 10 minutes.” Azure Service Health alerts now push directly to their ops channel so regional incidents are visible before customers start complaining.
Lessons That Stuck
Six hours of downtime is a brutal teacher. Here’s what we took away – and what any team running Azure infrastructure should internalize before something breaks.
Single-region is not a cloud architecture. It’s on-prem with more steps. If your workload matters, it lives in at least two regions with defined, tested failover. Full stop.
Availability Zones exist for a reason. Microsoft provides them at no extra charge for the architectural choice itself. You pay for additional VM instances if you need redundant compute, but the zone distribution costs nothing extra. There’s no excuse for skipping them on production workloads.
Azure Service Health is the first place you look, not the last. Build it into your incident runbooks. I’ve watched engineers spend 45 minutes debugging application code during an Azure infrastructure incident they could have identified in 30 seconds with one browser tab.
Resource group design is infrastructure design. One group for everything is technical debt that compounds fast. Separate environments, apply locks, enforce tagging policies from day one – not after your first incident teaches you why it matters.
The broader organizational challenges that come with cloud adoption extend well beyond Azure’s specific architecture. The Challenges in Cloud Computing: Security, Data Management & How to Overcome Them article covers a lot of the gaps that surface when teams move fast without thinking through the foundations first.
And if backup is still an afterthought in your Azure environment? Fix that now. Geo-redundant storage, snapshot schedules, and tested restore procedures are non-negotiable. The What Is Veeam Cloud Connect? A Complete Guide for IT Teams article is worth reading if you’re still working out your cloud backup strategy.
The Real Problem Was Never Azure
Here’s my honest take: most Azure failures I’ve seen aren’t Azure’s fault. They’re the result of teams treating cloud migration as a destination rather than an ongoing set of architectural decisions.
You can build something fragile in Azure just as easily as you can build something resilient. The platform hands you the tools – availability zones, geo-replication, Traffic Manager, Azure Monitor, ARM templates – but it doesn’t make you use them correctly. That part is on you.
The fundamentals matter. Understanding how geographies, regions, and availability zones relate to each other isn’t trivia. It’s the difference between a 6-hour outage at midnight and an automated failover that nobody notices. Subscriptions, resource groups, billing scopes – these aren’t administrative details. They’re the skeleton your architecture runs on.
If your team is heading into an Azure migration and you want someone with real post-mortem experience in the room, reach out to the SSE team. We’d rather help you avoid the 11 PM calls than help you recover from them.
