The Ticket Said the Server Was Slow
The ticket said the Citrix session host was sluggish around 4 PM every weekday. Twenty minutes with Process Explorer running and a baseline export from the day before, I had my answer: a line-of-business app was leaking around 40 MB per hour in Private Bytes vs Working Set, while the working set stayed almost flat. Task Manager showed nothing alarming. The customer’s monitoring tool was watching the wrong counter.
That engagement is the reason I tell every junior engineer the same thing: if you only look at one memory number, you will misdiagnose half your performance tickets. Working set and private bytes measure different things, and confusing them sends you down the wrong layer of the stack.
Two Counters, Two Questions
Private Bytes answers one question: how much memory has this process committed for its own use that cannot be shared with any other process? It includes the stacks, the heaps, raw virtual allocations, page tables, and the read/write portions of image and file mappings. Mark Russinovich documents this clearly in the Sysinternals Administrator’s Reference, and the value is backed either by physical RAM or by the page file.
Working Set answers a different question: how much of that process’s virtual memory is currently resident in physical RAM right now? It excludes Address Windowing Extensions regions and large page regions, but otherwise it is the slice the kernel decided to keep in RAM at this instant. The two values can diverge wildly, and that divergence is where the diagnostic signal lives.
Why the Difference Matters
A process can have 2 GB of Private Bytes and a 200 MB working set. That is not a problem on its own. The memory manager has trimmed the pages it judged cold and pushed them to the paging file. They will soft-fault back into the working set if the process touches them again.
The opposite case is more interesting. A process with steadily climbing Private Bytes and a stable working set is the classic leaked-but-untouched allocation pattern. The application is allocating and never freeing, but it is not actively reading the leaked regions, so the working set looks fine. Monitoring only RAM usage will miss this until the paging file fills up or commit limit is hit.
Reading Process Explorer’s Columns
Open Process Explorer, then View → Select Columns → Process Memory. Turn on these four at minimum: Private Bytes, Private Delta Bytes, Peak Private Bytes, and Private Bytes History.
The Private Delta Bytes column shows positive or negative change since the last refresh. A continual rise across refreshes is the textbook indicator of a memory leak. Peak Private Bytes records the largest commit the process reached since it started, which is useful when you walk up to a host an hour after the incident. Private Bytes History gives you a per-process sparkline; widen the column and you get a longer window.
For Working Set analysis, add the WS Private, WS Shareable, and WS Shared columns. WS Private is the part of the working set that is unique to this process. WS Shared is mapped pages like DLL code that other processes also map. When you are asked “how much RAM is this app really using?”, WS Private is the honest answer.
The Heat Map Trick
Process Explorer’s heat map columns shade cells by intensity. Sort by Private Bytes descending, and the worst offenders glow brightest. On a terminal server with 80 active sessions, this collapses a five-minute hunt into a five-second scan.
When to Reach for VMMap
Process Explorer tells you which process is bleeding memory. VMMap tells you which allocation type inside that process is responsible.
VMMap’s window has three bar graphs. The first is committed virtual memory. The second is the Private Bytes summary, broken down by allocation type — heap, stack, page table, private data, image, and mapped file. The third is the working set, showing how much of each type is resident in RAM. The colored segments are proportional, not positional; they tell you the mix, not the layout. For the actual address layout on a 32-bit process, use the Address Space Fragmentation dialog.
On the Citrix ticket I mentioned earlier, VMMap showed the leak living in the native heap, not managed heap. That ruled out a .NET garbage collection problem and pointed straight at a C++ component the vendor had updated two weeks before. Vendor patched it inside ten days once we sent the VMMap snapshot.
Tracking Leaks Over Time
Process Explorer is a live view. For trending across hours, switch to Performance Monitor with the Process object counters: Private Bytes, Working Set, Working Set – Private, and Page File Bytes. The Page File Bytes counter shows virtual memory the process has reserved in the paging file, and it tracks closely with Private Bytes minus the resident portion.
For deeper tracking, the Debug Diagnostic Tool (DebugDiag) from Microsoft will capture stack traces on each allocation. Configure a leak rule targeting Private Bytes or Virtual Bytes, let it run during the failure window, and it dumps the call stacks of the allocators. The two memory dimensions that matter for user-mode leak analysis are Private Bytes (heap) and Virtual Bytes (VirtualAlloc). DebugDiag handles both.
Windows Performance Recorder for the Hard Cases
When DebugDiag’s overhead is unacceptable on a production host, switch to Windows Performance Recorder from the Windows Performance Toolkit. WPR captures an ETW trace with allocation events that you analyze offline in Windows Performance Analyzer. The overhead during capture is much lower, but the analysis is heavier on the engineer.
An Opinion You Will Disagree With
Working Set is the wrong metric for capacity planning on Windows servers. I have watched too many teams size memory based on Task Manager’s “Memory (Active)” column and then get surprised when the host swaps under load. Private Bytes plus a buffer for the shared working set across processes is closer to what you actually need.
The argument against this is that committed memory is not the same as used memory, and over-provisioning costs money. Fair. But on a VM that hosts a managed service for a paying customer, I would rather waste 4 GB of RAM than explain to that customer why their app started paging at 9 AM Monday. VMware’s documentation on memory reservations agrees on this point for production workloads.
A Caveat About x64 Processes
Modern x64 processes with Control Flow Guard support reserve a 2 TB region for the CFG bitmap. The Virtual Size column will show numbers above 2 TB, which alarms people the first time they see it. Almost none of that 2 TB is committed, so Private Bytes is unaffected. Do not chase that number — it is reserved address space, not memory pressure.
Similarly, large-page allocations and AWE regions are excluded from the working set counter. SQL Server using locked pages will show a working set much smaller than its real RAM footprint. Cross-check with the SQL Server: Memory Manager counters before concluding the database is undersized.
What I Run on Every Engagement
Before I declare a server healthy after a memory incident, I run the same five-step check. We rolled this into the standard runbook across three client environments last quarter and it has caught two leaks before users noticed.
First, Process Explorer sorted by Private Bytes, heat map on. Second, Private Delta Bytes watched for sixty seconds — anything continuously positive without a workload reason gets investigated. Third, Performance Monitor’s Process(*)\Private Bytes counter logged for an hour to disk. Fourth, VMMap snapshots of any suspect process, taken five minutes apart. Fifth, a check of the system commit charge against the commit limit — if you are above 80 percent, you have a different problem than a single leaky process.
This is the same diagnostic pattern I document in our internal automation playbook, similar to the approach in running PowerShell across 200 servers when you need the same data set from every host in a farm.
Trade-offs Worth Naming
Process Explorer is free, low overhead, and ships nothing back to a vendor. That is why it is on every server I touch. The trade-off is that it is a point-in-time tool. You see now, not last Tuesday at 3 AM when the incident actually happened.
Performance Monitor data collector sets fix the historical problem but produce binary logs that nobody opens unless an incident forces them to. DebugDiag and WPR fix both problems but add measurable overhead and require expertise to interpret. The honest answer on most engagements is: run Process Explorer continuously on the desktop of the engineer on call, run a PerfMon data collector set 24/7 against the Process object, and reach for DebugDiag only when you have a confirmed leak and a maintenance window.
None of this matters if the host itself is at risk. Memory leaks that fill the page file can corrupt application state on a hard crash. Make sure the workloads on these servers are protected by reliable bare metal restore coverage before you start poking at production memory in the middle of business hours.
Practical Takeaway
Private Bytes tells you what a process has committed. Working Set tells you what is resident. A leak shows up as a steady climb in Private Bytes with a flat or slow-growing working set. Use Process Explorer to find the process, VMMap to find the allocation type, and DebugDiag or WPR to find the call stack. Reference the SANS Windows forensics material if you want to take this further into incident response territory.
If your environment is showing the symptoms in this article and you do not have time to chase them down internally, contact our team and we will run the five-step check against your hosts and tell you exactly which process is bleeding.
