Troubleshooting IIS

I recently had the pleasure if misdiagnosing a problem and spending 2 days in the deep weeds of IIS.  Figured I'd share my learnings and increase google fodder.

Update.20210713: Some wonderful info here! https://www.leansentry.com/Guide/iis-logs?source=guide-b
Update.20210505: If you are seeing 503 errors, don't forget that ELBs (AWS load balancers) can throw a very similar 503 error page...you can validate by browser inspecting the headers.
Update: Another finding, '500 0 0' entries that don't work with Failed Request Tracing should be dropping stuff in the Application log.

1. Figure out how to reproduce the error, maybe write a test? :)
2. Gather the specific error code:  "500" is the request status code, you also have subStatus codes and win32-status codes.  e.g. 500.1.1, 403.1.2
   1. If you have server access, set the hosts file on the server to point www.website.com to 127.0.0.1, then load a browser on the server itself and re-create the error - moar info
      1. This also usually yields actual error messages (which are generally also dumped into the Application event log)
   2. You can change a web.config setting for show remote errors (I've had so-so luck with it in the past)
3. Check the event log
   1. Usually in the Application log you'll see ASP.NET errors/warnings
   2. You can filter System for: Source: HttpEvent
   3. AppPool recycles are logged
   4. IISRESETs are logged
4. Check the IIS logs
   1. C:\inetpub\logs\LogFiles\W3SVC1 (corresponds to IIS site id #)
   2. Each line should have 500 0 1 or something matching the actual error code
      1. 2016-08-18 00:00:08 172.17.51.124 GET /keepalive.aspx - 443 - 172.17.8.25 ELB-HealthChecker/1.0 - 200 0 0 0
         1. See the end?  200 is the status code, 0 is substatus, 0 is win32-status, 0 is bytes
         2. Win32 codes: https://msdn.microsoft.com/en-us/library/ms681381.aspx
5. Check the appPool status (asp.net crashes can leave it in a stopped state, or Rapid Fail Protection could be kicking in, perhaps the default 30hour recycle is going on?)
   1. Bad code is usually the cause of legit asp.net crashes.  Ensure the code has logging on crashes.
6. White screen of death.  (user only sees a white screen, nothing else, fast load time)
   1. Test locally on the server.  Does it work?
   2. Are you running a load balancer?
   3. Are any instances in the pool?
   4. No?
   5. Because you didn't account for a change that broke the health check?
   6. Fix the health check!
7. You are deep in the weeds now.  Here there be dragons.
8. Check HTTPERR logs: C:\Windows\System32\LogFiles\HTTPERR
9. You might see something like: 2016-08-17 02:50:07 172.17.8.25 14728 172.17.51.124 80 HTTP/1.1 GET /keepalive.aspx 503 1 N/A DNN
   1. 80 HTTP/1.1 GET /keepalive.aspx 503 1 N/A DNN
      1. port, protocol, request type, request path, status code, sub-status code, REASON, appPool
         1. N/A - A service unavailable error occurred (an HTTP error 503). The service is not available because an internal error (such as a memory allocation failure or URL Reservation List conflict) occurred.
10. If you are seeing 503 errors, you are really in the weeds.  http.sys is the thing that handles basic request routing for the appPools, and 503 means "the system...is down...the system...", i.e. the appPool isn't even seeing the reqs.
11. Check here for a list of 503 REASONS: https://support.microsoft.com/en-us/kb/820729
    1. Boiled down:
       1. Bad configuration file XML
          1. web.config
          2. C:\Windows\System32\inetsrv\config
          3. C:\Windows\Microsoft.NET\Framework64\v4.0.30319\CONFIG\machine.config
       2. Overloading of the server
       3. Identity/permissions (appPool identity, file perms, combo of both)
       4. URL reservation issues or IP inclusion list
       5. appPool issues
12. Failed Request Tracing
    1. If you rely on IIS, you probably should use this...it's not too bad once you get over the gotchas
13. If URL reservation/IP inclusion list issues:
    1. netsh http show servicestate
    2. netsh http show urlacl
14. Other random stuff to try if you're this desperate.  These were pulled as common comments 'solutions'.
    1. Commenting out the IPv6 entry in hosts (wat)
    2. Enable 32-bit on the appPool (you coded something wrong/badly)
    3. aspnet_regiis.exe /i  (this is a horrible solution)
    4. Starting the stopped appPool (you would be amazed how often this came up)

Code Snippet to test web.config (PowerShell)

#I did not write this, one of those links' authors did.

Import-Module WebAdministration
$site = "DNN" #or whatever

try
{
 Get-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -location "$site" -filter "system.webServer" -name . | Out-Null
 Write-Output "All is Well"
}
catch [System.Exception]
{
 Write-Output $_ 
}

Some reading

http://mvolo.com/where-did-my-iis7-server-go-troubleshooting-503-quotservice-unavailablequot-errors/
http://geekswithblogs.net/leesblog/archive/2010/12/09/503.1-service-unavailable-error-resolution.aspx
http://haacked.com/archive/2007/05/18/service-unavailable-errors-in-iis-7-are-killing-me.aspx/
https://support.microsoft.com/en-us/kb/832975
https://blogs.msdn.microsoft.com/webtopics/2010/02/17/a-not-so-common-root-cause-for-503-service-unavailable/
http://lupolt.net/?p=96
https://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/55f71614-ef1b-4015-b9c8-a42c1e700c25.mspx?mfr=true
http://www.zerrouki.com/recycling-applicationpool-if-503-error-is-detected-in-http-sys-error-log/