The alert came in at 2:47 AM. File shares encrypted. Domain controllers unreachable. The backup server — also encrypted. The client’s IT lead called it a total loss. Twenty minutes into triage, I found the one thing that saved them: backup encryption with certificate-based keys, configured eight months earlier during a routine hardening engagement at SSE.
The Incident Timeline
The attack started with a phishing email opened on a workstation at 11:14 PM. Within 90 minutes, lateral movement had reached the backup server via SMB. By 2:47 AM, all primary storage was encrypted and a ransom note was waiting on every desktop.
The backup server ran Veeam Backup & Replication. The local repository was encrypted by the ransomware — it had write access via an over-provisioned service account. The offsite immutable copy on S3-compatible object storage was untouched. Object lock policies set at the bucket level blocked every overwrite and delete attempt, regardless of credential validity. The ransomware had network access to the bucket endpoint. It still couldn’t touch the data.
Recovery came from the 48-hour-old immutable restore point. RTO was roughly six hours for critical systems. Not ideal, but survivable.
Root Cause Analysis: Three Failures, Not One
Post-incident forensics identified three compounding failures. We followed the same methodology detailed in our Digital Forensics for Incident Response field guide — collect, preserve, then analyze.
Failure 1: Unencrypted Local Backups
The local backup repository had no encryption. Once ransomware reached that host, it encrypted the backup files directly alongside primary storage. The service account had read/write access to both environments — separation existed on paper, not in practice.
For SQL Server environments, this failure shows up constantly. When you enable Transparent Data Encryption (TDE), SQL Server automatically encrypts all backups — a fact that surprises most DBAs. If TDE isn’t in use, you can still encrypt backups explicitly with a server certificate. Using dbatools, a PowerShell-based SQL Server administration module, the command looks like this:
Backup-DbaDatabase -SqlInstance sql01 -Database ProductionDB `
-EncryptionAlgorithm AES256 `
-EncryptionCertificate "BackupEncryptCert" `
-BackupDirectory "\\nas01\backups"The certificate lives on the SQL Server instance. A backup written with that certificate cannot be restored anywhere else without the matching cert and private key. Even if an attacker grabs the .bak file, it’s useless without the key material. Store the certificate export offline, separately from the backup files themselves.
Failure 2: Over-Provisioned Service Account
The Veeam service account held local admin rights on every protected server plus direct write access to the backup repository share. Default in a rushed deployment. Nobody had ever applied least privilege. When the ransomware operator had those credentials, they had everything needed to destroy the local backup copy.
The fix is structural. Separate the backup operator role from the backup repository write role. The account running backup jobs should not have direct filesystem access to the repository. Veeam’s built-in RBAC model supports this cleanly. In Kubernetes environments, Veeam Kasten takes this further — it binds IAM roles directly to Kubernetes RBAC, preventing lateral escalation between the backup namespace and application namespaces. Ransomware that compromises one namespace can’t pivot to destroy backup artifacts held in another.
Failure 3: No Anomaly Detection on Backup Infrastructure
Nobody was watching the backup server. No alerting on mass file enumeration, no monitoring for encryption events. When integrated with S3-backed backup targets, Amazon GuardDuty detects and flags suspicious access patterns — including malicious IP attempts and anomalous access to bucket resources. That layer wasn’t configured. It is now.
The Remediation Architecture
We rebuilt the backup architecture around three constraints: encryption at every layer, immutability enforced at the storage platform, and zero implicit trust between backup infrastructure and production systems. The NIST Zero Trust Architecture (SP 800-207) framing applies directly here — treat backup infrastructure as its own trust boundary. Access flows one direction: backup jobs pull from production. Production never reaches into backup storage.
For the Windows-side hardening baseline, we applied the same controls documented in Windows Server Hardening: Stop Guessing, Start Baselining. Backup servers are production servers. They get the same treatment.
The rebuilt configuration:
- Local repository: AES-256 encrypted, service account scoped to write-only, no read access from the production network segment
- Offsite copy: S3 object storage with 30-day immutability lock, separate IAM credentials stored offline and not present on any domain-joined system
- SQL Server: TDE enabled on all production databases, certificate exports stored in offline key management separate from backup media
- Monitoring: GuardDuty enabled on all S3 backup buckets, alerting on more than 100 object read requests per minute from any single source IP
Lessons Worth Keeping
My position: an unencrypted backup stored on a domain-joined server is not a backup. It is a second copy of data with an identical attack surface. If ransomware can reach your primary storage, it can reach that server. Encryption and immutability are the only controls that survive credential compromise.
One caveat that gets glossed over: immutable backups are not a substitute for restore testing. We’ve taken over environments where immutable copies existed but had never been validated. Incomplete application consistency, misconfigured retention policies, silent corruption — any of these can make a technically intact backup unrestorable under pressure. Test quarterly, at minimum. Document the results.
The Veeam Ransomware Trends Report 2024 puts the probability of being hit at 75%, with most organizations reporting more than one incident. Kubernetes environments aren’t exempt — Red Hat’s State of Kubernetes Security Report 2023 found that 67% of organizations delayed Kubernetes deployments specifically over security concerns. Backup encryption and immutability apply to containerized workloads the same as traditional infrastructure. Veeam Kasten handles this at the namespace level, with end-to-end encryption of application data and metadata both in transit and at rest, enforcing consistent security rules across on-premises, public cloud, and hybrid deployments.
If you’re not certain your backup architecture would survive a ransomware event, contact SSE for a backup architecture review. We run these as standalone engagements — no full audit required to get a clear answer.
