NullReferenceException at Scaleout.Collections.RecentDictionary

[NadavB]

Hi,

In the last 2 weeks we noticed that from time to time we have picks of a few minutes of NullReferenceException errors that come from inside the scaleout client. It doesn't happen every day, but sometimes it happens a few times a day.

System.NullReferenceException "message":"Object reference not set to an instance of an object.\
at Scaleout.Collections.RecentDictionary`2.MoveToFrontOfLru(Node node)\
\
at Scaleout.Collections.RecentDictionary`2.Find(TKey key, Boolean updateLru)\
\
at Scaleout.Collections.RecentDictionary`2.TryGetValue(TKey key, TValue& value)\
\
at Scaleout.Client.InProc.LruEvictionCache`1.Get(ObjectKey& key)\
\
at Scaleout.Client.Cache`2.CheckInProcCaches(ObjectKey& objectKey, ReadOptions options, Boolean lockObj)\
\
at Scaleout.Client.Cache`2.ReadInternal(ObjectKey objectKey, ReadOptions options, ReadFlags flags, LockToken lockToken, AccessOperation traceOp, TKey key, CancellationToken ct)\
\
at Scaleout.Client.Cache`2.Read(TKey key, ReadOptions options, CancellationToken cancellationToken)\


We use ScaleoutClient nuget version 3.1.4 and Scaleout.Collections nuget version 1.0.3, .Net Framework 4.7.2.
What can we do about it?

Thank you

 

[markw]

Hi Nadav,

Thank you for reporting this and supplying the stack trace. We'll investigate and will let you know what we find.

 

[markw]

Hi Nadav,

We've reviewed error you reported and haven't yet located the source of the problem.

Please copy the Scaleout.Collections.pdb file from the attached zip to the web server(s) that are experiencing this issue--it should be copied to the same folder as the Scaleout.Collections.dll file in your application's directory. When the error occurs again, these debug symbols will allow the stack trace to print the exact line number in the MoveToFrontOfLru() method, which will help our engineering team narrow down the root cause.

Also, does this exception get thrown soon after your application starts up (for example, right after an IIS app pool recycle)? Or does it occur after the app has been running for a long time?

Thanks!

 

[markw]

Hello again Nadav,

We found and resolved an issue in the Scaleout.Collections NuGet package that may explain the exception that you reported. Please update your project to use version 1.0.4--we'll be interested to hear if this resolves the issue.

Thanks!

 

[NadavB]

Hi Mark,

Thank you for the quick answer and for the help. I was waiting for it to reproduce after I added the pdb file, and here is the stack trace with the line numbers:

"System.NullReferenceException","priority":11,"message":"Object reference not set to an instance of an object.\
\
at Scaleout.Collections.RecentDictionary`2.MoveToFrontOfLru(Node node) in D:\\\\Users\\\\markw\\\\dev\\\\Github\\\\ScaleoutCollectionsDotNet\\\\src\\\\Scaleout.Collections\\\
ecentDictionary.cs:line 303\
\
at Scaleout.Collections.RecentDictionary`2.Find(TKey key, Boolean updateLru) in D:\\\\Users\\\\markw\\\\dev\\\\Github\\\\ScaleoutCollectionsDotNet\\\\src\\\\Scaleout.Collections\\\
ecentDictionary.cs:line 451\
\
at Scaleout.Collections.RecentDictionary`2.TryGetValue(TKey key, TValue& value) in D:\\\\Users\\\\markw\\\\dev\\\\Github\\\\ScaleoutCollectionsDotNet\\\\src\\\\Scaleout.Collections\\\
ecentDictionary.cs:line 931\
\
at Scaleout.Client.InProc.LruEvictionCache`1.Get(ObjectKey& key)\
\
at Scaleout.Client.Cache`2.CheckInProcCaches(ObjectKey& objectKey, ReadOptions options, Boolean lockObj)\
\
at Scaleout.Client.Cache`2.ReadInternal(ObjectKey objectKey, ReadOptions options, ReadFlags flags, LockToken lockToken, AccessOperation traceOp, TKey key, CancellationToken ct)\
\
at Scaleout.Client.Cache`2.Read(TKey key, ReadOptions options, CancellationToken cancellationToken)\


Do you think that the issue that you found and resolved is the same as the issue that we have?

Thanks again!

 

[markw]

Thank you. The line number appears related to the fix we made, so I think there's a very good chance it's resolved.

I cannot say it's fixed with 100% certainty because we still aren't able to repro this particular exception--the error here is a result of corruption that occurred in the cache at an earlier time. The issue we fixed could manifest in many ways, depending on your app's unique client cache usage pattern (how objects are added, accessed, removed, and evicted), which is not something we can duplicate in our testing.

What I can say with certainty is this: Version 1.0.4 of Scaleout.Collections is better and more reliable than 1.0.3, and we recommend using it under any circumstances, even if this particular exception isn't resolved.

Also, I've attached debug symbols for the new 1.0.4 release. This will be helpful to us if the error persists or behavior changes.

 

[NadavB]

Ok I'll update the Scaleout.Collections to 1.0.4 and I'll keep tracking it. Thank you!

 

[markw]

Ok I'll update the Scaleout.Collections to 1.0.4 and I'll keep tracking it. Thank you!

Hi Nadav, just a one-week check-in: have you had an opportunity to deploy 1.0.4 and determine whether it helped?

 

[NadavB]

Hi Mark, it will probably take more than a month until it will be deployed. I'll keep you updated.