Thursday, June 5, 2008

Recommendations for SharePoint Application Pool Settings

Talking to a buddy on Friday, the post he said he actually read was one I must have posted on 2003. I guess he's not necessarily a frequent reader, but it made me realize it's a topic I haven't touched in a very long time. Shane was also asking me about this. There are some good descriptions of the settings (MSDN IIS Settings Reference), but what I'll give you here is my thoughts on the settings with recommendations. I do recommend reading about the IIS 6 features which list out very well the enhancements and control for your application pools there's some really good info. Let me first talk about 3 app pools for which I have very specific guidance. Then get into the real guidance for the content web application app pool settings. If I don't mention a setting it's because I expect it to be off or set to default. Let me know if I leave anything out or if you have additional guidance.
IIS 6/SharePoint 101
Web Site = IIS Virtual Server = SharePoint Web Application
Web Applications have Application Pools... You can manage your web apps in IIS manager.
Application Pools have Worker processes... You'll see a section in your IIS manager for managing your app pools.
Worker Processes are processes that consume memory and CPU... You'll see them as w3wp.exe (Don't get confused here. By default there is a 1:1 for Application pool to worker processes.
Default Application pool
Just leave it alone, don't use it, and occasionally make sure nothing is using it. Do NOT delete it. Why... Cause IIS doesn't like it when it doesn't exist. You'll find IIS gets mad if this gets deleted. Even if you plan to never use it, just leave it alone. Don't even rename it. You can put something funny in the description field to remind you, but as was the unspoken best practice in the IIS 4 and IIS 5 days with the default web site. Only newbies use it, the more experienced web admins create new web apps and stop or deleted it. Now I'm saying though you may want to delete it. Don't.
If you're sure nothing is using this app pool, there is one setting you can enable to have it shut down if for some reason it ever spins up.
Performance Application Pool Settings
Shudown the application pool after it's been idle for 10 minutes. The first check box is the key to these app pools.
Health Application Pool Settings
Pinging. Turn it off if it's on since this app pool doesn't really need to run except for a short period of time during administration.
Rapid failure. Make sure it's not enabled at all.
Central Admin App Pool, SSP Admin App Pool
Recycling Application Pool Settings
Once per night around 2-4 A.M is typical, but even that shouldn't be necessary if it is getting shut down when not in use (see peformance). Recycling every 120 minutes is not a good idea and is completely unnecessary.
Max Physical and Virtual memory: These app pools typically don't take much memory, and my point with these is to simply let them run when they need to. If you wanted to limit the memory, I'd limit it to 500MB on a typical system. If you need to conserve, you could limit these guys to 200MB. They seriously shouldn't need much memory.
Performance Application Pool Settings
Shudown the application pool after it's been idle for 10 minutes. The first check box is the key to these app pools. You really don't need for these app pools to be running all the time since they are really only used for administration. If you find them cold and slow you could use a warm up script, but on most systems for administration you shouldn't really care since it's only you and your ops folks using it. If you're feeling technical you could shut down this app pool worker process when you're done with it to reclaim memory, but if it's set to shut down when idle as suggested, it's a no brainer to just let it do it's thing.
Don't worry about the CPU. If you do care you can limit it to 5%, but really you're just changing your admin settings... getting in and getting out. Likely if this is a big operations environment you're going to be doing this during a maintenance window anyway.
Web Gardens. Don't even think about using them here. You don't even want the one worker process running very long.
If you're really trying to preserve memory you could really share the central and SSP admin app pools, meaning you go in and configure the SSP to use the Central admin app pool, but only if the SSP & Central administration is done by a common team. For security reasons if you have 2 different groups managing the different interfaces you wouldn't want to do this. Since I'm not listing the steps here, if you don't know how to do this, don't worry about it. Just keep them both shut off when you're not using them and it won't matter.
Health Application Pool Settings
Pinging. Turn it off if it's on.
Rapid failure - uncheck it. Any web applications using this app pool could fail if this gets disabled.
Startup/Shutdown. You don't want these app pools creating unnecessary errors when they start up a bit slow. So I'd recommend increasing the limits for startup and shutdown to 300 each.
Identity Application Pool Settings
This is where your domain account would be for a farm environment. A unique account for these app pools is recommended. You don't need to manage the accounts here since you can either manage it via STSADM or the web UI. If the central admin one goes bad, this might be the first place you'd look if you can't get into anything. Do refer to the KB on resetting your password. In SPS 2003 you would have spent some time on this tab if you ever had to reset your passwords or domain accounts. They've made it easier if you know the commands.
SharePoint App Pool or %Your Content App Pool% (Whatever you call it)
Recycling Application Pool Settings
Worker process in minutes (off) - keep it off
Worker process recycle at a specific time. Once per night around 2-4 A.M is suggested. Recycling every 120 minutes is not a good idea and will cause problems if it happens that often. I do recommend the nightly app pool recycle. If you don't like the time of night you can change it, but I do recommend keeping it different than all other app pool times. So 1:05 is ok on one and 1:25 on another for example. If you turn it off, you're really not taking advantage of reclaiming memory. It's way too common to see people not clean up their objects properly closing, deleting just doesn't happen like it should and many expect .NET to clean up for them. Sad. The nightly recycle is handy to get you back your RAM. I don't recommend doing it more often than once a night. I have heard of 2 like once right at the end of the day and one at the beginning of the day. You know, after the backups, profile imports, crawls, and whatever else went on during the night. I think recycles during the user load (during the day or whenever load is near it's peak) is not recommend is a bit clunky. It is possible to push a warm up to pre cache your most common pages right after a cycle, but not necessary in most environments.
Max Physical and Virtual memory: This setting is a lot of control here. This section is for recycling application pools which consume too much memory. Focusing on physical I typically like to limit app pools around 800MB to 1200 MB max on a 32 bit app with very few app pools depending on the number and amount of memory. On a server with 2 GB RAM I'd set it at around 800MB max. On a 4GB of RAM server around 1GB and more if more with a max around 1200. On a 64 bit web front end with 8-16 GB memory I've heard of settings of 2GB of RAM or even allowing it to let it ride, rather than limiting it. You really need to profile it since these can really grow to process and cache. The greater the amount of memory and the greater the load the higher the worker process will grow. When people ask about configuring the app pool, this is where they are usually asking what the numbers should be. What you are doing here is explicitly limiting the app pool from consuming more memory. Notice this setting is on the recycle tab, there's a reason for that. When an app pool reaches the max it isn't like the max processor setting. It will cycle the worker process which is like a tiny reboot or similar to an iisreset, but not since sometimes we want this to happen so we can release our memory. You really don't want to cycle more than a couple of times per 24 hour period in an ideal world. I've heard of some trying to cycle right before the morning peak occurs so they have the most amount of memory available, then a cycle right at the end of the day before the backups or crawling begins.
Performance Application Pool Settings
Idle timeout - off (unchecked) - Don't shut it down unless you have some web apps that aren't used very often. It will help you reclaim memory if that's the condition. If you have one main web app for example, just leave it off or uncheck it.
Request queue - unchecked
CPU limits - unchecked. You want to allow these main worker processes for the app pool to use the CPU.
Web Gardens. Ok. So this is controversial, more so than the recycling even. It was tough explaining to some that IIS recycles are a feature not a cop out. Web Gardens now feel a bit like that same conversation. Let me start with nearly all environments should NOT need them ever. Even some product and support people would prefer not to see people use these.
From MSDN:
"Because Web gardens enable the use of multiple processes, each process will have its own copy of application state, in-process session state, caches, and static data. Web gardens should not be used for all applications, especially if they need to maintain state. Be sure to benchmark the performance of the application before deciding whether Web garden mode is appropriate.
When using a Web garden, it is important to understand how session state and round robin works. It is also important to consider of how other application pool settings affect the application."
My experience has been that with another worker process I can get more performance out of a single web application. Troubleshooting is more difficult, and process isolation is difficult. Developers don't like em either. So before I say never use em. I say, go with the recommendation listed above. A web garden should always be compared against a baseline. Some may find a web garden gives them better performance. Since this is so controversial, I say, if you are doing your performance tests and trying to squeeze out more perf, start with one, then retest with 2. If you don't see much of an improvement, don't use it. If you find more than 15-20% out of a second worker process then you might find it worth it. It is easy enough to take it back to one. The more CPUs the more likely you will get something here. Feel free to share your comments in the comments section on this one. Man some people are really passionate one way or another about these.
Health Application Pool Settings
Pinging. Keep this enabled. You could ping it every 10-15 minutes this will keep it running and keep it alive if you want it to monitor the app pool to check it to start it if it is found stopped. You don't want to mix the monitor pinging and the stop when idle. If you want it to run turn off stop when idle, and visa versa if you want it to stop, don't ping monitor it.
Rapid failure - uncheck it unless there is only one web application using this app pool and all content is on this app pool. (Joel's recommendation) I'm not a fan of this one. It should help when you're doing isolation to shut down poorly performing or worker process cycling, but doesn't help when you have one main web application with one app pool. You decide if you want to go against this. I'm really not a fan of this setting for SharePoint Applications or in any hosting situation. I've been burned by these default settings. I hate finding a server that's shut down because of some silly startup failure of one worker process that causes everything to go down. I'd rather have a chance to troubleshoot with a warning than finding it all down and finding out that it was this setting that shut everything down.
Startup/Shutdown. You don't want these app pools creating unnecessary errors when they start up a bit slow. So I'd recommend increasing the limits for startup and shutdown to 300-900 each. You should never get close to this. That's a good thing. The more web apps you have, the longer it will take. If you want it to shut down for starting up slow you can reduce these settings. This doesn't dictate how long it will take.
Identity Application Pool Settings
This is where your domain account would be for a farm environment. A unique account for these app pools is recommended. You don't need to manage the accounts here since you can either manage it via STSADM or the web UI. If the central admin one goes bad, this might be the first place you'd look if you can't get into anything. Do refer to the KB on resetting your password. In SPS 2003 you would have spent some time on this tab if you ever had to reset your passwords or domain accounts. They've made it easier if you know the commands. If you ever are on this tab, its to verify that it's not using a local account.
What if I have a bad app pool or web app?
If you have an web application you're trying to isolate, put it in its own app pool, then limit the CPU, limit the memory and cycle often (like every 120 min) and shut it down (using the settings above) when it's not in use. What this will do is limit the exposure of the memory from the other web applications. There are graceful ways of cycling a specific worker process. You can use these same methods for determining for troubleshooting which worker process that's taking up tons of RAM or cycling often what needs to be reconfigured or split out and isolated. Let the developer know his app is in the isolation box and he needs to clean it up before you give it more memory and stop cycling it so much. :)
Can I consolidate Web Application app pools?
Yes you can. If you have a lot of web apps you can use common app pools. You'll be smarter about the memory you consume. If you think one is taking up too much RAM consolidate the good ones and isolate the bad one like I suggest for bad ones.

Web Gardens:

My experience has been that with another worker process I can get more performance out of a single web application.

"This will never be the case, especially in a .NET environment, which SharePoint lives in. The problem here is two fold. 1) Memory and 2) context switching due to inter-process communication of the worker processes. 1) You use more memory with web gardens and it’s especially hurtful in a .NET world because you will load the entire framework in *each* worker process, so you will immediately lose at least 100MB for eash additional worker process you spin up as part of the web garden. 2) Since you have multiple worker processes for a single application pool, you now have extra inter-process communication that must take place and also numerous more context switches that must occur. Both of these, while increasing concurrency, ultimately decreases overall performance as well as requests/sec. the box can fulfill.

The *only* case I have ever run across in which using web gardens was desirable was when the customer’s w3wp.exe’s were consistently reaching the x86 2GB process limit size and would crash. In this case, we turned on web gardens, set the number of flowers (aka worker processes) to 3, which then split the memory load among the three, thus solving the upper limit 2GB process size crash we were experiencing."