I’m taking a short break from Hangfire series, but I will get back to it.
This time - Where did my memory go ? Or to be more exact: Why is this using so much memory?
The story starts with one IIS application pool using around 6 Gigabytes of memory on one of our test environments. It was several times above the values that we expected it to use, so we decided to investigate.
Without much thinking we fired up Visual Studio installed on the test server, and attached to the process. Since the application was build in Debug mode we had all the pdb files in the website folder.
Do I have your attention now? The above paragraph is of curse a joke and a bunch of anti patterns. Don’t do any of them!
Now that I have your attentions let’s get back to what really happened. Since it was a remote sever handling multiple enviroments the only sane step to take is to make a memory dump of the process, kill it and analyze it on a developers machine. Since we always have Sysinternals Suite in
c:\tools (highly recommend having it, because when You need it the most, there is no time for download). Sysinternals Suite contains a simple console application for making process dump called
ProcDumpis a very powerful tool and it’s capabilities go far beyond what I will show now. To just give a glimpse flags that control then the dump will be triggered:
cl- create a dump when CPU threshold is above or below given limit
e- create a dump when unhandled exception is thrown
h- create a dump when processes window is not responsive
ml- create a dump when CPU threshold is above or below given limit
pl- create a dump when a given performance counter is above or below a given limit
t- create a dump when the process terminates
- absence of this flags means trigger the dump now For a better insight of what ProcDump can do fire it up without any arguments and it will display a very good help.
In this case we wanted to get the dump right now, so the command line looked like this:
procdump.exe 25944 c:\temp\ -ma
25944is the process id. We could pass a process name, but since it is a website hosted by IIS its process is named w3wp.exe.
Process id (PID in short) can be obtained be many tools such as
ProcessExplorerwhich is also a part of SysInternals Suite. But the default Task Manager available on any Windows machine also has it. Just right click on the Name column, and check
PIDin the menu that will appear.
c:\tempis the folder where the dump will be written to
maflag says that we want a full dump with all of processes memory written to the file.
If you get this error:
Error opening w3wp.exe (25944): Error 0x00000005 (5): Access is denied.
That is because it has to be run with administrator privileges like any debugger. The normal output should look like this:
ProcDump v8.0 - Writes process dump files Copyright (C) 2009-2016 Mark Russinovich Sysinternals - www.sysinternals.com With contributions from Andrew Richards [12:48:59] Dump 1 initiated: c:\temp\w3wp.exe_160917_124859.dmp [12:49:03] Dump 1 writing: Estimated dump file size is 5612 MB. [12:49:59] Dump 1 complete: 5614 MB written in 55.6 seconds [12:50:00] Dump count reached.
So we have the dump. What next? This is the topic for the next post :)