Supporting/Troubleshooting StateServer remote clients - bunch 'o questions

[YetAnotherTechMonkey]

Hi all, I'm not a Dev, more ops supporting SOSS that I inherited, so jumping into this pretty blind. (Any existing posts/documentation that I missed, feel free to post.)

I've got one StateServer supporting 3 environments (remote agents), it was working fine, but yesterday one Test env stopped working. Unfortunately it's being used by a proprietary framework so I don't have much visibility, but main part of error is:

'ScaleoutServiceUnavailableException'><message><![CDATA[Unable to open a connection to the local ScaleOut service. Verify the local service is started and joined

I've compared to other environments, re-deployed (known working) code, uninstalled and re-installed, rolled back OS patches, upgraded agent... Not sure where to go from here. Some questions that came up trying to figure things out:

1 - How do you verify remote client connection? On the (IIS) web servers I can open the management console, see the "Local Store" (that's not local, but on the server cluster), and see the IP's of the other environments in the remote-clients tab, but none from the the broken environment. Additionally, can use get-cache and see a couple appid/cachename pairs on the expected server hosts. On the remote agent servers, where/what is the actual process/.exe? On the server, finding the various SOSS services is easy. On the remote clients - I can't see anything in services, task manager, anything. So I have no way of restarting the agent/service without rebooting the whole OS.

2 - What's the agent < - > server compatibility story? Can I upgrade agents or server first and the other will be backward compatible? (Just enough for testing, then upgrade the rest.)

3 - The online docs (http://static.scaleoutsoftware.com/docs/soss_help/content/index.html) mention windows event logs, but there's no mention of the actual logs, esp remote agents. I finally found the soss_tlog0.txt log, which repeatedly says 'Cannot open parameters file path C:\Program Files\ScaleOut_Software\StateServer\soss_params.txt' - why is it trying to read that rather than client_params? is that by design?

4 - Additionally, we've had some issue w/ old apps not playing well with TLS 1.2 - disabling 1.0 and 1.1 on the windows level (Registry/Group Policy) breaks some things. Has anyone had issues or experience w/ this for SOSS?

 

[markw]

How do you verify remote client connection?

Opening up the ScaleOut Management Console was a good first check--it verified that your remote client machines can reach the ScaleOut service over the network.

However, the Management Console only connects to the ScaleOut service's management port. Client applications connect to the service's server port, so you'll want to make sure that your remote clients have that port value configured correctly.

On the remote client machine, open up the ScaleOut Management Console and click on the Client Configuration node in the tree on the left. In the configuration screen that appears, take a look at the "Server Port" values in the list of Gateways:

(thumbnail, click to expand)

You'll want to make sure that (1) these are all the same value and (2) that they all match the value of the server port settings in the Host Configuration screen of the host(s) in your server cluster:



Also, you'll want to make sure that there are no firewall rules that would prevent your application from connecting to that server port. You can use PowerShell's Test-NetConnection cmdlet on the remote client system to check connectivity to the server port:

Test-NetConnection 127.0.0.42 -Port 721

If there's a connectivity problem then that'll be the cause of the exception you're seeing. (You only need to be concerned about TCP connectivity to the server port--no need to worry about UDP or ICMP.)

On the remote agent servers, where/what is the actual process/.exe? On the server, finding the various SOSS services is easy. On the remote clients - I can't see anything in services, task manager, anything. So I have no way of restarting the agent/service without rebooting the whole OS.

There is no separate agent process on remote client machines. Any application that connects to the ScaleOut service uses client libraries that are loaded directly into the process. So let's say your remote client is running an ASP.NET web app behind IIS; to restart everything, you'd just need to recycle the IIS app pool(s) instead of rebooting the OS.

2 - What's the agent < - > server compatibility story? Can I upgrade agents or server first and the other will be backward compatible? (Just enough for testing, then upgrade the rest.)

Yes, client libraries and servers are generally compatible within a major release.

I finally found the soss_tlog0.txt log, which repeatedly says 'Cannot open parameters file path C:\Program Files\ScaleOut_Software\StateServer\soss_params.txt' - why is it trying to read that rather than client_params? is that by design?

Yes, that's by design. The presence of that file tells our client libraries whether or not the ScaleOut Service is installed locally.

The soss_tlog0.txt files are really only meant for ScaleOut support staff when troubleshooting. There can be a lot of opaque diagnostics in those files that make them tough to interpret.

4 - Additionally, we've had some issue w/ old apps not playing well with TLS 1.2 - disabling 1.0 and 1.1 on the windows level (Registry/Group Policy) breaks some things. Has anyone had issues or experience w/ this for SOSS?

Disabling those older protocols should have no effect on the ScaleOut service. The product uses OpenSSL for encrypted connections instead of the built-in Windows Schannel stuff, and it only supports TLS 1.2 for client-to-server communications.

 

[YetAnotherTechMonkey]

Thank you! In this particular case, the underlying issue was a breaking change in the framework, but I needed the info in your reply for troubleshooting. (Might be worth considering if some of this should be more explicit/easier to find in the documentation.)

...Also, you'll want to make sure that there are no firewall rules that would prevent your application from connecting to that server port. You can use PowerShell's Test-NetConnection cmdlet on the remote client system to check connectivity to the server port:

Test-NetConnection 127.0.0.42 -Port 721

If there's a connectivity problem then that'll be the cause of the exception you're seeing. (You only need to be concerned about TCP connectivity to the server port--no need to worry about UDP or ICMP.)

Did you perhaps mean to test against localhost while on the server, then from the remote-client *to* the server? On the remote client web servers, hitting 127.x any of the ports 721-724 (We're using SSL) doesn't "succeed". Against the server, of course, it does (720-724). But that's to be expected, agents connecting to a port on the server, not hosting/accepting any connections.

There is no separate agent process on remote client machines. Any application that connects to the ScaleOut service uses client libraries that are loaded directly into the process. So let's say your remote client is running an ASP.NET web app behind IIS; to restart everything, you'd just need to recycle the IIS app pool(s) instead of rebooting the OS.

Yes! That was one of the key things I was missing, thank you.

Yes, that's by design. The presence of that file tells our client libraries whether or not the ScaleOut Service is installed locally.

The soss_tlog0.txt files are really only meant for ScaleOut support staff when troubleshooting. There can be a lot of opaque diagnostics in those files that make them tough to interpret.

Disabling those older protocols should have no effect on the ScaleOut service. The product uses OpenSSL for encrypted connections instead of the built-in Windows Schannel stuff, and it only supports TLS 1.2 for client-to-server communications.

All very useful.

Two more operational questions if you don't mind:

1 - Does that single client config .txt file set everything needed? I can deploy an MSI well enough, but would like to use azure pipelines (or other config management) to do the configuration.

2 - Along those lines, can they handle hostnames or any plans for that? (We have to manually confgure unicast) What about IPv6?

 

[markw]

1 - Does that single client config .txt file set everything needed? I can deploy an MSI well enough, but would like to use azure pipelines (or other config management) to do the configuration.

Yes, the soss_client_params.txt file contains all the connection information used by remote clients to connect.

We recommend deploying ScaleOut remote client components via the MSI package whenever possible--there are native dependencies and registry entries that make manual installation an error-prone hassle. So your best bet would be to deploy the MSI and then copy over the default soss_client_params.txt file with a preconfigured one.

(That said, we have a new client library that doesn't require an installer. More on that below...)

2 - Along those lines, can they handle hostnames or any plans for that? (We have to manually confgure unicast) What about IPv6?

The soss_client_params.txt file does not support hostnames or IPv6. We do offer hostname support, however, in the newer Scaleout.Client NuGet package. The Scaleout.Client library supports more modern deployment techniques since it has no external dependencies and lets you specify connection info through your app’s config file. It’s a significantly different API, however, so it may not be a good fit if you’re supporting a legacy project that’s already invested a lot of code in our traditional APIs.