Soss.Client.NotReadyException: The server is not yet ready to perform this operation.

[ridopark]

Hi,

We are seeing this exception shown below. It seems to happen more often when SOSS is used for the first time after reboot, but it happened at other times too. Could you please tell us what's the recommended way to get around this? I added some retry logic that would retry after sleeping one sec, but sometimes it can take more than 5 retries to get past this.

System.TypeInitializationException: The type initializer for 'App.StateServer.StateServerClient' threw an exception. ---> Soss.Client.NotReadyException: The server is not yet ready to perform this operation.
   at App.StateServer.StateServerClient.SossClientLogin() in C:\src\test\StateServer\StateServerClient\Client.cs:line 1506
   at App.StateServer.StateServerClient..cctor() in C:\src\test\StateServer\StateServerClient\Client.cs:line 1048
   --- End of inner exception stack trace ---
   at App.StateServer.StateServerClient.ReadObject(Application application, UInt16 index, SossKey key)
   at ConsoleApp1.Program.Main(String[] args) in C:\src\test\ConsoleApp1\Program.cs:line 33

Thank you

 

[markw]

Hi,
Is this exception coming out of a call to LoginManager.getInstance().Login(...)?

If so, the most likely cause is that the back-end auth provider (a custom executable that you've deployed to your ScaleOut server machines) is returning an error when invoked by the ScaleOut service. If this happens at times other than startup, it may also be returning errors when a session needs to be re-authenticated.

Does your custom auth provider have any logging, and, if so, do you see error entries that align with this exception?

 

[ridopark]

The custom executable I'm using is a Powershell script that just returns "permissions 3 timeout minutes 60" for debugging purposes. I wouldn't expect there would be any error returned by the Powershell script. Another weird thing about this is that this never happened when I use local SOSS server and local clients. It started when we deployed this to QA environment where we have multiple client machines and two SOSS servers.

I'm also seeing a lot of this exception, "Soss.Client.NotAuthorizedException: User is not authorized to perform this operation.". I'm guessing this exception is thrown because the auth did not happen since it may have just moved on after retrying auth for x number of times due to "Soss.Client.NotReadyException: The server is not yet ready to perform this operation."

We are using v5.12.3.378. Is there any known issue with this version?

 

[markw]

There aren't any known issues with auth in 5.12.3.378. There are a few things you'll need to check:

1. From an administrative command prompt, please run: soss.exe show_ns_auth

In the output, make sure that the cache(s) that you're using are listed, and verify that the path to your powershell script that's shown there is valid on both ScaleOut servers.

If there's anything wrong with a cache name or path, you can use the set_ns_auth subcommand to fix it. For example:

soss.exe set_ns_auth name="My cache name" method=custom path="C:\MyScripts\MyTestScript.ps1"

(Corrected on 9-July-2024; you have to invoke powershell in the path argument...)

soss.exe set_ns_auth name="My cache name" method=custom path="powershell -F C:\MyScripts\auth.ps1"

2. Please try running your powershell script manually from the command prompt on *both* of your ScaleOut hosts, using the full path that's listed in the soss.exe show_ns_auth output. Verify that your "permissions 3 timeout minutes 60" output appears on both machines.

...It's easy to forget to set the PowerShell execution policies on a new Windows server and then hit a security exception the first time you run an unsigned ps1 script. (And when you call Set-ExecutionPolicy, be sure the -Scope is set to the default of "LocalMachine" since the script may not run under your same user account when the ScaleOut service invokes it.)

3. If your script path is OK and you've verified that it runs on both servers: Please gather and zip the soss_tlog0.txt files from both of your ScaleOut hosts (located in the installation folder at C:\Program Files\ScaleOut_Software\StateServer). Send them to [email protected] and reference this forum thread. Our team will review them to learn about this error.

 

[ravikc]

Hi Mark,

I am running into this issue as well. Is it possible for you to provide us a sample custom authorization provider script? I want to compare to see if I am doing what I am supposed to do in the script. Currently it just writes "permissions 3 timeout minutes 60" into the console for successful authorization. Is this the right thing to do?

Thank you.

 

[markw]

Hi,
I've tested successfully with a one-line script that looks like this:

# auth.ps1
Write-Output "permissions 3 timeout minutes 60"

Use the soss.exe command to register this custom auth provider for a namespace:

soss.exe set_ns_auth name="My cache name" method=custom path="powershell -F C:\MyScripts\auth.ps1"

A couple of important notes:

• My earlier example in the post above (#4) didn't use "powershell -F" in the path argument. Turns out this is required, otherwise Windows won't execute the script.
• Make sure that your PowerShell execution policies are configured to allow scripts to run on all of the machines in your farm.

 

[vasantbala]

I attached logs over the email (with subject: log files to troubleshoot login issues) yesterday. Please take a look and advice. This is the only issue that is stopping us to proceed further.