Tuning SOSS for peak demands
- Thread starter keith.dorre
- Start date
Hi Keith,
Assuming you're running on Windows with an ASP.NET app, the two most commonly tuned parameters are:
1) max_client_cache in the soss_params.txt file: This setting adjusts the size of the internal cache of deserialized objects within the .NET client libraries--the libraries basically hold on to references of recently requested objects. So when you read a big object, this client-side cache dramatically reduces response time by eliminating the network and deserialization overhead. It may be that you have too many large objects for this in-process cache to operate effectively, so increasing its size may help.
By default the max_client_cache is set to 100000KB (100 MB), so try increasing this if you have memory to spare on your web servers. Changes to the soss_params.txt file will take effect when the ScaleOut StateServer service is restarted, and your app pools will then pick up the change from the SOSS service the next time they're recycled.
Also, if you haven't already, consider configuring your load balancer to use session affinity. If a user's requests always (or, at least, usually) hit the same web server then it can make better use of the client cache described above. Using affinity isn't strictly required by SOSS, but if a user's web requests always go to the same web server then the odds of a client cache hit will improve, which can reduce serialization/network overhead.
To measure whether these changes have an effect, consider monitoring the "ScaleOut StateServer Service\Client Cache Read Hits" performance counter in Windows perfmon before and after you make the change.
2) max_svr_conn in the soss_client_params.txt file: If your web servers only have ScaleOut client libraries installed and are accessing remote, dedicated SOSS machines over the network then increasing max_svr_conn can increase throughput (assuming you have network bandwidth available). This setting is used to increase the number of simultaneous connections that your web servers have open to remote SOSS servers (if you run the SOSS service directly on your web servers then this setting will have no effect). By default the value is 4, but it can be increased to 32 simultaneous connections.
Changing values in the soss_client_params.txt will take effect the next time your app pools are recycled.
Having said all this, we're really only taking a stab in the dark here with so little information... if you can pinpoint a bottleneck on a specific resource (Network? CPU? ...which process?) then we may be able to offer more targeted advice.
Assuming you're running on Windows with an ASP.NET app, the two most commonly tuned parameters are:
1) max_client_cache in the soss_params.txt file: This setting adjusts the size of the internal cache of deserialized objects within the .NET client libraries--the libraries basically hold on to references of recently requested objects. So when you read a big object, this client-side cache dramatically reduces response time by eliminating the network and deserialization overhead. It may be that you have too many large objects for this in-process cache to operate effectively, so increasing its size may help.
By default the max_client_cache is set to 100000KB (100 MB), so try increasing this if you have memory to spare on your web servers. Changes to the soss_params.txt file will take effect when the ScaleOut StateServer service is restarted, and your app pools will then pick up the change from the SOSS service the next time they're recycled.
Also, if you haven't already, consider configuring your load balancer to use session affinity. If a user's requests always (or, at least, usually) hit the same web server then it can make better use of the client cache described above. Using affinity isn't strictly required by SOSS, but if a user's web requests always go to the same web server then the odds of a client cache hit will improve, which can reduce serialization/network overhead.
To measure whether these changes have an effect, consider monitoring the "ScaleOut StateServer Service\Client Cache Read Hits" performance counter in Windows perfmon before and after you make the change.
2) max_svr_conn in the soss_client_params.txt file: If your web servers only have ScaleOut client libraries installed and are accessing remote, dedicated SOSS machines over the network then increasing max_svr_conn can increase throughput (assuming you have network bandwidth available). This setting is used to increase the number of simultaneous connections that your web servers have open to remote SOSS servers (if you run the SOSS service directly on your web servers then this setting will have no effect). By default the value is 4, but it can be increased to 32 simultaneous connections.
Changing values in the soss_client_params.txt will take effect the next time your app pools are recycled.
Having said all this, we're really only taking a stab in the dark here with so little information... if you can pinpoint a bottleneck on a specific resource (Network? CPU? ...which process?) then we may be able to offer more targeted advice.
Thanks for the reply and such a considered one with the limited info provided. You are right we are a windows shop with ASP etc.
We went away and did some deep dive investigation into our load balancing strategy and and the read hits counter, as suggested. For historical reasons - no one is actually quite sure, some architect may have had a preference - we aren't using Sticky Sessions (affinity). We then looked at the perf counters for read hit and during load misses were out of the park. Based on this our approach is going to be in two phases, firstly change the load balancers to have sticky - reducing and then look at the max_client_cache.
So we are changing the LB's tomorrow, then monitor the perf counters. and then we will look at the max_client_cache - From our current monitoring the system has roughly between 50 - 60 % extra capacity at peak load - can we roughly assume doubling the max_client_cache from 100000 to 200000 be safe ? or is there a formula where we can be more accurate?
Thanks again
K
We went away and did some deep dive investigation into our load balancing strategy and and the read hits counter, as suggested. For historical reasons - no one is actually quite sure, some architect may have had a preference - we aren't using Sticky Sessions (affinity). We then looked at the perf counters for read hit and during load misses were out of the park. Based on this our approach is going to be in two phases, firstly change the load balancers to have sticky - reducing and then look at the max_client_cache.
So we are changing the LB's tomorrow, then monitor the perf counters. and then we will look at the max_client_cache - From our current monitoring the system has roughly between 50 - 60 % extra capacity at peak load - can we roughly assume doubling the max_client_cache from 100000 to 200000 be safe ? or is there a formula where we can be more accurate?
Thanks again
K
Great, it sounds like you're taking the right approach by just changing one thing at time and then measuring the impact.
Doubling the client cache size to 200000 should be fine since you presumably have several gigs of free RAM. But do keep in mind that the max size of that in-process cache is a rough estimate/heuristic... there isn't a fast, reliable way to find the exact size of an arbitrary .NET object at runtime, so the ScaleOut client libraries use the size of your objects when serialized as an estimate of their runtime memory usage. This estimate can be inaccurate if your objects serialize very efficiently--I mention this not because I expect it to be a problem, but simply because sometimes users are surprised if changes in w3wp memory usage don't match up perfectly with changes to max_client_cache.
Doubling the client cache size to 200000 should be fine since you presumably have several gigs of free RAM. But do keep in mind that the max size of that in-process cache is a rough estimate/heuristic... there isn't a fast, reliable way to find the exact size of an arbitrary .NET object at runtime, so the ScaleOut client libraries use the size of your objects when serialized as an estimate of their runtime memory usage. This estimate can be inaccurate if your objects serialize very efficiently--I mention this not because I expect it to be a problem, but simply because sometimes users are surprised if changes in w3wp memory usage don't match up perfectly with changes to max_client_cache.