2012R2, IIS, and KB3052480 - random ASP.NET app domain restarts

I will leave out the 'how we came to this conclusion' part, but suffice to say it took 3 people the better part of 2 full days.

There is a, in my humble opinion, rather gaping bug in 2012R2 IIS that manifests itself like this:

  • You have many file changes in the web root, but they are contained to the 'safe' directories (i.e. not in /bin or whatnot)
  • In theory, IIS doesn't care and you go about your merry way - because it only monitors /bin and /app_data (iirc?) and some config files
  • You find yourself chasing spikes in your (New Relic) data, traces that make no sense (the transactions that get called the most have a lot of looong traces)
  • Nobody understands what's going on, because no errors are being thrown
You can verify that you're seeing a lot of ASP.NET app domain restarts by looking at perfmon:
ASP.NET Applications - (your instance) - Application Lifetime Events

What you should be seeing upon a legit recycle is one startup event and one shutdown event, and possibly a 'compilation started' event.

What happened in our case is that (we suspect) Launch Darkly log files were being written to a lot (which is fine), but IIS was detecting these as configuration changes.  We found that by adding a <healthMonitoring> rule in: C:\Windows\Microsoft.NET\Framework64\Config\web.config

<add name="Application Lifetime Events Default" 
 eventName="Application Lifetime Events" 
 provider="EventLogProvider" 
 profile="Default" 
 minInstances="1" 
 maxLimit="Infinite" 
 minInterval="00:01:00" 
 custom="" />

Do an iisreset and you'll see log start to come into the application event log.

EventID: 1309
Source: ASP.NET 4.0.30319.0
Event code: 1002
Event messsage: Application is shutting down. Reason: Configuration has changed.
Event detail code: 50004

A lot of googling/digging later, we turned up this hotfix: https://support.microsoft.com/en-us/kb/3052480

After applying the hotfixes and rebooting the servers (yes, it prompts you to restart even though the documentation states otherwise), we have seen ZERO app domain restarts.

\o/

Unexpected ASP.Net application shutdown after many App_Data file changes occur on a server that is running Windows Server 2012 R2


Consider the following scenario:
  • You have a server that is running Windows Server 2012 R2.
  • You set up a website on Internet Information Services (IIS).
  • You create a simple website that creates or deletes lots of files in the App_Data folder.

In this scenario, you experience an unexpected ASP.NET application shutdown because of high disk I/O and configuration changes.

Comments

Popular posts from this blog

DFSR - eventid 4312 - replication just won't work

Fixing duplicate SPNs (service principal name)

Logstash to Nagios - alerting based on Windows Event ID