Our monitoring board looked fine at 11 PM. By 3 AM, we had 5,000 queued connections and a site returning 502 errors to every visitor. That night became our real education in Nginx server optimization – the kind you only get by actually breaking things in production.
The Incident: A Site That Stopped Responding
We were running a mid-size e-commerce platform on a four-core VPS with NGINX 1.24.0 in front of a PHP-FPM backend. Traffic was unremarkable – nothing unusual about the load leading up to 2:47 AM on a Tuesday in October 2023.
Then response times climbed from 80ms to 12 seconds in about four minutes. Then the site stopped responding entirely. Requests were piling up in the NGINX queue. PHP-FPM workers were maxed out. Users got timeouts or 502s. All from an outage we had no warning about.
We had not deployed anything in 72 hours. That was the part that made it worse.
Timeline: From First Alert to Full Recovery
Here is exactly what happened:
02:47 – Uptime monitor fires. Response time exceeds 10-second threshold.
02:52 – On-call engineer connects. NGINX error log shows repeated upstream timeout errors.
03:04 – PHP-FPM status page confirms all 50 workers saturated. Queue depth: 1,200+.
03:11 – Decision made: restart PHP-FPM to clear the queue. Site recovers briefly.
03:19 – Same problem returns within eight minutes of recovery.
03:31 – I get pulled in to focus specifically on the NGINX configuration layer.
04:14 – Configuration changes deployed and tested. Site stable. Queue draining.
Forty-seven minutes from my first look to resolution. But understanding the why took another week.
Root Cause Analysis: Three Mistakes That Compounded
This is what post-mortems are actually for. Not the fix – the understanding.
We found three separate configuration problems, each manageable alone, but together they caused cascading failure when traffic spiked slightly above normal. Compound failures like this show up across technologies – a similar pattern turned up in this Windows Group Policy Incident: A Real Post-Mortem where independently minor misconfigurations combined into something serious.
Problem 1: worker_processes was set to 1. Our server had four CPU cores. We had never changed the default. One NGINX worker was handling every connection. When it got busy, everything queued behind it.
Problem 2: keepalive_timeout was set to 75 seconds. The default value. Never questioned. Slow clients were holding connections open for over a minute each. With enough concurrent users, we ran out of available connections before the queue could clear.
Problem 3: NGINX was buffering large PHP responses in memory. Some of our pages generated heavy JSON payloads. NGINX held those in memory buffers while slowly sending them to clients on slower connections. Memory pressure started affecting worker performance.
We also had no configuration snapshots from the past six months. Keeping backups in a separate location – and that includes web server config files, not just application data – would have given us an immediate rollback path if any of our emergency changes had made things worse.
None of these three problems would have caused an outage on a quiet night. Together, during a modest traffic spike (a referral link had gone semi-viral on Reddit without anyone noticing), they compounded into a full service failure.
Nginx Server Optimization: The Config Changes That Fixed It
Worker Processes and Connection Limits
First change. Should have been done on day one.
# Before
worker_processes 1;
worker_connections 512;
# After - matches actual CPU core count
worker_processes auto;
worker_connections 1024;
Setting worker_processes auto tells NGINX to spawn one worker per CPU core. On our four-core machine, that is four workers instead of one. Connection handling capacity effectively quadrupled with a single line change.
Keepalive Timeout and Request Limits
We dropped the keepalive timeout aggressively:
keepalive_timeout 15; # Was: 75 (stock default)
keepalive_requests 100; # Added - was never configured
This works well for high-concurrency public-facing sites. But not for APIs where clients maintain persistent long-lived connections – you will need to profile your own traffic before applying these values.
Buffering and Backend Connection Pooling
We adjusted buffer sizes to stop accumulating large payloads in memory:
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 32k;
And for the PHP-FPM upstream, we added a keepalive pool so NGINX reuses backend connections instead of opening a fresh one per request:
upstream php_backend {
server unix:/var/run/php/php8.2-fpm.sock;
keepalive 32;
}
Every change was tested by spinning up a clean test environment first and running Apache Bench against it before touching production. (The first time around we skipped this entirely and pushed changes at 3 AM. Ugly. Do not do that.)
Lessons Learned – and What We Automated After
Here is what I took away from this:
Default configs are not optimized for your workload. NGINX defaults are conservative and safe, not performant. Opinion: if you are deploying NGINX and not reviewing these settings against your actual hardware and traffic profile, you are running at risk you do not know about yet.
Post-incident automation is the best automation. Within a week I had a PowerShell script running a health check against every new NGINX deployment – worker count, keepalive values, buffer settings – flagging anything that still looks like a stock default. Saves roughly 20 minutes per deployment and has caught three configuration oversights since October 2023.
Monitoring without baselines is just noise. We had uptime monitoring. We had no performance baseline to compare against. Now we capture a metrics snapshot after every deployment and alert when response times drift more than 20% from that baseline. The same config-drift mindset from Ansible Automation: Server Management and Patching at Scale translates directly to web server tuning across a fleet of hosts.
And the uncomfortable truth: most Nginx server optimization problems I have seen are not sophisticated tuning challenges. They are default values nobody questioned. The fix is usually five lines of config. The hard part is building the discipline to check before things break at 3 AM.
If your infrastructure team wants help reviewing web server configurations, building automated deployment checks, or working through a post-mortem from your own outage, get in touch with the SSE team – we help organizations stop reactive firefighting and start building infrastructure that holds up when traffic gets unpredictable.
Related reading: Windows Group Policy Incident: A Real Post-Mortem | Threat Hunting Techniques: A SOC Readiness Audit | Ansible Automation: Server Management and Patching at Scale
