ScaleOut StateServer Performance Monitoring

[Arbiter-Bryan]

I'm interested in monitoring the performance of our ScaleOut StateServers (v5.10.5).
I know I can view some stats with Windows PerfMon.

Which stats are going to indicate an issue?
What should I be watching for?
And most importantly, how do I get those stats off the host and into a log aggregator like DataDog?

 

[markw]

Hi Bryan,
Yes, the service's PerfMon counters will be the best approach on Windows. I don't have experience with DataDog, but a quick Google search shows that its agent can consume Windows performance counters.

At a high level, monitoring the ScaleOut StateServer Service\Host Joined counter will tell you if the ScaleOut service is running and active (it'll have a value of 1). You can set up an alert if that counter drops to zero (indicating that the host has dropped out of the store).

Beyond that, there are some counters that will tell you about the load and performance of the service--what's "normal" there will vary depending on your particular application, your load, and your environment. You might consider simply capturing some of the other counters for a while (CRUD rates, for example) to get a feel for what's normal, and, later on, set up alerts on them once you understand their expected range.

 

[Arbiter-Bryan]

DataDog can collect custom Windows Performance Counters over WMI. It looks like the counters can be found with a bit of PowerShell:

PS C:\Windows\system32> Get-WmiObject -Query "select * from Win32_PerfFormattedData_sossperflib_ScaleOutStateServerService"

__GENUS                           : 2
__CLASS                           : Win32_PerfFormattedData_sossperflib_ScaleOutStateServerService
__SUPERCLASS                      : Win32_PerfFormattedData
__DYNASTY                         : CIM_StatisticalInformation
__RELPATH                         : Win32_PerfFormattedData_sossperflib_ScaleOutStateServerService=@
__PROPERTY_COUNT                  : 47
__DERIVATION                      : {Win32_PerfFormattedData, Win32_Perf, CIM_StatisticalInformation}
__SERVER                          : ComputerName
__NAMESPACE                       : root\cimv2
__PATH                            : \\ComputerName\root\cimv2:Win32_PerfFormattedData_sossperflib_ScaleOutStateServerService=@
AvgBytesRead                      : 290
AvgBytesWritten                   : 7612
AvgGeoServerBytesReceived         : 0
AvgGeoServerBytesSent             : 0
Caption                           :
ClientCacheReadHits               : 871
ClientConnections                 : 54
ClientCreates                     : 7
ClientDeletes                     : 0
ClientReads                       : 913
ClientUpdates                     : 196
Description                       :
DistributedCacheReadMisses        : 200
EventsFired                       : 0
EventsNotFired                    : 22
EventsNotHandled                  : 0
Frequency_Object                  :
Frequency_PerfTime                :
Frequency_Sys100NS                :
GeoServerDeletesReceived          : 0
GeoServerDeletesSent              : 0
GeoServerPushEligibleObjectshost  : 70444
GeoServerPushEligibleObjectsstore : 212183
GeoServerTotalQueueLength         : 0
GeoServerUpdatesReceived          : 0
GeoServerUpdatesSent              : 0
HostCreatesPersec                 : 25
HostDeletesPersec                 : 30
HostJoined                        : 1
HostReadsPersec                   : 806
HostReady                         : 1
HostUpdatesPersec                 : 221
LockCollisions                    : 0
LocksAcquired                     : 18
Name                              :
NumberofJoinedHosts               : 3
ObjectMemorybytes                 : 727115243
Objects                           : 70444
ParallelMethodInvocations         : 0
PercentLoad                       : 33
PostedEvents                      : 0
Queries                           : 0
RemoteClients                     : 4
SingleMethodInvocations           : 0
Timestamp_Object                  :
Timestamp_PerfTime                :
Timestamp_Sys100NS                :
PSComputerName                    : ComputerName