SSH with Channel issues

Discussion of open issues, suggestions and bugs regarding network security and data protection solution - SecureBridge
ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

SSH with Channel issues

Post by ED-Clint » Tue 05 Jun 2018 09:11

Hi,

I am still working on establishing stable SSH tunnels and I have found something rather odd. The connection, I am not sure if the issue is the channel or the ssh connection, seems to drop or stop in some way after a random amount of time.

I say it like this because I never see an OnDisconnect event from the Channel or an OnError, nor do I see a BeforeDisconnect/AfterDisconnect event from the client.

I thought that maybe the MySQL/MSSQL queries that I send down the tunnels were having some sort of issue, or the server side, but I can't get any response out of the tunnel other than a failure to connect or a timeout from the query software.

If I restart the SSH connection and Channel it all then works again, for a time.

I have set the following on the ScSSHClient;

Options.ServerAliveCountMax := 3;
Options.ServerAliveInterval := 18;
Options.TCPKeepAlive := true;
Timeout := 90;

and the follwoing on the ScSSHChannel;

Timeout := 60;

I do not have any limits set at the server end (that I know of).

I have a test program that I could send you, it presented errors after nearly 5 hours of running (a simple query fires every 10 mins) but on a customer site I have seen this happen after about an hour consistently.

I have run other SSH tunnel software on the same PC connecting to the same server doing the same job and it maintains the tunnel all day.

I am fairly certain that I am simply doing something wrong but at this point I am not sure where else to look.

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Tue 05 Jun 2018 12:14

2nd test, almost 5 hours later it fails.

None of the disconnect or error event messages have appeared.

Windows shows that it is still listening on that port;

TCP 127.0.0.1:778 0.0.0.0:0 LISTENING

The SSH server still shows the connection as active. Yet my MSSQL query received this error when the MSConnection tried to connect;

05/06/2018 12:47:29 Error: TCP Provider: An existing connection was forcibly closed by the remote host.

Client unable to establish connection because an error was encountered during handshakes before login. Common causes include client attempting to connect to an unsupported version of SQL Server, server too busy to accept new connections or a resource limitation (memory or maximum allowed connections) on the server.

I restart the Channel and SSHClient and it now works again.

What is happening that causes the Channel and SSH connection to remain open as far as the Server is concerned yet does not let the MSConnection use it any more?

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Tue 05 Jun 2018 14:04

I changed the timer to run the query every 10 seconds, it just counts the rows in a table. The table is not used by anything else. The reply is a 4 digit number.

Every single time, attempt 32 fails. I have run this from 2 different PC's creating tunnels to this same MSSQL server and running this query, both of them fail on attempt 32.

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Tue 05 Jun 2018 14:35

It looks like this is an issue with number of concurrent connection and the reason I should of set Pooling := true on the MSConnector!

I can now get past 32 attempts and I'll leave this running to see how long the connection stays up.

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Wed 06 Jun 2018 13:27

My test ran all night over 5600 times without issue.

However when I then used the test program on the customers server connecting to the same database server (MSSQL 2017) sending all connections over the SSH tunnel it took just over an hour for the same thing to happen.

All connections have pooling := true; the number of established connections shown via netstat was only 3-6 at any one time. The tunnel and the SSH connection never errored or disconnected but the MSSQL server sarted refusing connections.

The server does not limit concurrent connections and over this period never had more than ~90 connections at a time.

Running an SSH Tunnel demo from /n Software runs all day without issue. So although the issue looks like it is at the server end and it is certainyl the SQL server that starts to refuse connection I simply do not understand why this problem only occurs when using the Devart SDAC and Securebridge components.

Any help here would be greatly appreciated.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: SSH with Channel issues

Post by ViktorV » Wed 06 Jun 2018 13:30

Most likely, the cause of this behavior is the value of the global variable MAX_CONNECTIONS, which equals 32 by default. To solve the issue, please try adding the ScSSHChannel module to uses clause and set the global variable MAX_CONNECTIONS to the value you need.
For example:

Code: Select all

initialization
  MAX_CONNECTIONS := True;	

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Wed 06 Jun 2018 14:35

Thank you Viktor, I have set this to MAX_CONNECTIONS := 250; and am testing with this.

However in your example you have MAX_CONNECTIONS := True; does this mean I could set it to False and if so does this mean that there is no connection limit imposed by ScSSHChannel?

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: SSH with Channel issues

Post by ViktorV » Wed 06 Jun 2018 15:00

We apologize for an inaccuracy. The code should be:

Code: Select all

initialization
  MAX_CONNECTIONS := 250;

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Fri 08 Jun 2018 08:21

Forgive me Viktor but I could not find any documentation on that global variable.

Why does it exist?
Can I set it to 0 for unlimited or does there have to be a limit?
Is the limit against concurrent connections or against a counter while the Channel is in use?
If it is a counter, why? How can I set the counter to 0, if at all?
Do you have any documentation you could forward to me, or point me at if it is online somewhere?

Kind regards,
Clint.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: SSH with Channel issues

Post by ViktorV » Fri 08 Jun 2018 09:50

This global variable is nowhere described. Its value equals the maximum allowed number of connections established by TScSSHChannel. To set it to a maximum value, please set it to MaxInt. For example:

Code: Select all

initialization
MAX_CONNECTIONS := MaxInt;

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Thu 14 Jun 2018 10:40

I had the MAX_CONNECTIONS set to 250 and this gave me about 10 hours before it crashed and needed to be restarted. So I upped this to 5000 and it seemed to be ok.

I then turned this in to a service which on service start the ssh connection is established and the tunnel created, on server stop both are indonnected in order.

I then set a task schedule to restart the service (using NET STOp then NET START) every morning. So far so good.

The problem I have now is that by the morning the service has hung so it will not stop, I get the "The service cannot accept control messages at this time" message. So I have to taskkill the service and manually start it.

Why does the ssh connection/tunnel hang after a certain amount of time?
How can I monitor the tunnel and when it is not in use by anything restart it?
Do you have any furhter suggestions to keep a tunnel open for use?

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: SSH with Channel issues

Post by ViktorV » Mon 18 Jun 2018 09:51

In our testing, when using SSH tunnel we never encountered its freeze
The error causes in your program may be a lot, you should make a detailed logging of your program to detect the causes of its freeze. You can use the TScSSHChannel.OnSocketConnect and TScSSHChannel.OnSocketDisconnect event handlers to monitor the number of clients using your tunnel.

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Thu 05 Jul 2018 09:19

Thanks Viktor. Can you give me an example of using the TScSSHChannel.OnSocketConnect and TScSSHChannel.OnSocketDisconnect event handlers, because anything I do int hese event handlers results in an exception.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: SSH with Channel issues

Post by ViktorV » Thu 05 Jul 2018 12:19

In order for us to be able to give you a detailed answer, we need a sample demonstrating the behavior you mentioned. Therefore, please, compose a small sample demonstrating the use of the TScSSHChannel.OnSocketConnect and TScSSHChannel.OnSocketDisconnect event handlers that lead to exception and send it to us using the contact form https://devart.com/company/contactform.html.

ED-Clint
Posts: 36
Joined: Thu 18 May 2017 08:52

Re: SSH with Channel issues

Post by ED-Clint » Thu 05 Jul 2018 14:29

I'm fairly certain I am just using them incorrectly, I can prepare a test project to send to you and it will take a little time. However in the mean time can you please just give me an example of their use? I did not see this in your demo programs and it is likely all I need to get this working.

Post Reply