Strange slowness

RobV

New member
#1
I'm working on sharing session with an api (don't ask why), everything is working fine except when I do some load testing on it.

I have a single method that all it does is insert an object into session and return, this call is taking 200ms per call, I'm also running scale out host on my machine, but the time is the same for it deployed on server. If I switch to inproc session the time this takes become almost instant.

So I wrote, a second method that all it does is take the same object and inserts a 100 unique entries of it into session, the time this method takes is almost instant.

My question is, why is method 1 so much slower then inserting 100 unique objects into session?


C#:
//Single insert takes 200ms on call, switching to inproc method takes 6ms (this is only the time of the network round trip)
[HttpPost]
public async Task<IHttpActionResult> Insert(InsertObjectRequest request)
{
    HttpContext.Current.Session[request.Key] = request.Value;
    return Ok();
}

//100 unique inserts is instant takes < 10ms
[HttpGet]
public IHttpActionResult LoadTest()
{
    string value = "json object 42kb in size"

    var writes = new System.Diagnostics.Stopwatch();
    writes.Start();

    for (int a = 0; a < 100; a++)
    {
        //INSERT
        string sessionKey = Guid.NewGuid().ToString("N");
        HttpContext.Current.Session[sessionKey] = value;
    }

    writes.Stop();

    return Ok(writes.ElapsedMilliseconds);
}
 

markw

Administrator
Staff member
#2
Hi Rob,

Which session provider are you using? The traditional one that ships with the main product installer, or one of the newer NuGet providers (sync/async)?

Also, a few notes about this test (and session state in general):
  • Session data isn't written back to the ScaleOut service after your LoadTest() controller action completes; the entire Session dictionary is serialized at the very end of the request pipeline. So individual calls to the Session dictionary's setter aren't doing full round trips--you're just manipulating a local, in-memory dictionary there, which is why it seems so fast.
  • Yes, in-proc session state will be orders of magnitude faster than any out-of-process provider like ScaleOut or SQL Server. There's no network or serialization overhead... it's just manipulating an in-process data structure.
  • In any case, 200 ms does seem slow. A couple of possible explanations:
    • You're only measuring the first request after the app starts up (the first round trip is slow because of JIT'ing, three-way TCP handshakes when first connecting, etc.). Subsequent requests will be much faster.
    • You're running under a debugger (async operations in particular can be very slow while a debugger is attached)--the async session provider will definitely run slow while debugging.
    • request.Value is big (several megabytes).
 
Last edited:

RobV

New member
#3
Hi Rob,

Which session provider are you using? The traditional one that ships with the main product installer, or one of the newer NuGet providers (sync/async)?

Also, a few notes about this test (and session state in general):
  • Session data isn't written back to the ScaleOut service after your LoadTest() controller action completes; the entire Session dictionary is serialized at the very end of the request pipeline. So individual calls to the Session dictionary's setter aren't doing full round trips--you're just manipulating a local, in-memory dictionary there, which is why it seems so fast.
  • Yes, in-proc session state will be orders of magnitude faster than any out-of-process provider like ScaleOut or SQL Server. There's no network or serialization overhead... it's just manipulating an in-process data structure.
  • In any case, 200 ms does seem slow. A couple of possible explanations:
    • You're only measuring the first request after the app starts up (the first round trip is slow because of JIT'ing, three-way TCP handshakes when first connecting, etc.). Subsequent requests will be much faster.
    • You're running under a debugger (async operations in particular can be very slow while a debugger is attached)--the async session provider will definitely run slow while debugging.
    • request.Value is big (several megabytes).
Thanks for the reply Mark, that's good to know about session data and order of events. we are using the traditional one that ships with the product installer and gets installed in the gac.

I got the call down to about 150ms yesterday, but the 200ms was not on the initial request, that was much higher because like you said JIT.

It is under a debugger but I had tried running in release and on a server and didnt see much better behavior.

The request.Value is 42kb, the interesting thing I noticed yesterday was that if I restarted my local soss instance and started with everything zero'd out it would be about 15ms per call. As the memory used went up, the slower the inserts got, probably by the 3rd time the calls had crept up to 50ms per call.

Can you tell me about Soss.Client.DataAccessor.ConcurrentRequests and when it should be used, if this was going to be a high traffic api would I want to up this number? Whats the max it should be set at, any problems with upping this?
 

markw

Administrator
Staff member
#4
I'm afraid don't fully grasp the scenario here--does the slow performance occur only during load tests, or is it when you're manually poking at the site with a web browser (or is the site under other load while you're manually browsing)?

Overall memory usage shouldn't affect response times from the ScaleOut service (so long as you have enough RAM on your machine and the OS isn't paging to disk), but, again, I'm not clear on how your test works. So when you mention that "the memory used went up"... is that 42 KB object growing larger with every web request, or is something else consuming memory?



The DataAccessor.ConcurrentRequests property controls the size of a pool of unmanaged connection handles that sits between our native and managed client libraries. It defaults to 64 handles (it was 20 prior to version 5.0.6.206, circa 2012). The size of this pool doesn't need to be changed unless a client is running a huge multithreaded load while hitting a local instance of the ScaleOut service.

If you're accessing a remote ScaleOut service and foresee heavy load, you may want to increase the max_svr_conn parameter in your client machine's soss_client_params.txt file (located in the product's installation folder).
 
Last edited:

RobV

New member
#5
So what I’m seeing is on a single session for a user, if I insert 1 object that is 42k the response is quick. But the more objects that I insert into that session, say a 100, I see the insert speed slow. These are also 1 off inserts with unique keys to that session.

This is a poc that I’m doing to prove that what we are trying to accomplish will work in our environment and speed is a top priority.
 

markw

Administrator
Staff member
#6
I see, thanks for clarifying. This is a case of the user's session growing larger and larger--this test case will get slower and slower with any out-of-process provider because of how ASP.NET groups a user's session items into one object.

All the items in a user's session dictionary are serialized into a single blob at the end of the web request and then sent to the ScaleOut service. (Remember: it isn't the setter on the session dictionary that's doing the round trip.) So in your test's final request, the pipeline retrieves the whole dictionary of 99 items at the start of the web request and then sends back the updated dictionary with 100 items.

Since each item is 42KB, you'll have a 4.2 megabyte session dictionary by the 100th request, and that's transferred over the network twice for every web request: once for the session retrieval at the start and once for the session update at the end... so 100-200ms is roughly what I'd expect it to take to push 8.4 megabytes around on a gigabit network.

So the short answer is that you don't want to allow a user's session to have unbounded growth like this. Keep your session dictionary as lean as possible!
 
Last edited:
Top