UPDATED Please read: Host key signature failure error

[tcaduto12068]

Hi,
We updated the IBM system and we are still getting the error:
( I put some logging in the functions) It gets to the SyncKeyExchange and the other side rejects the
signature Sbridge is sending)

Is there a way to capture what it's sending to verify it's not changing the host key signature?

send init ok
process init ok
KeyExchangeAlgorithm 1 ok
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Connection was closed by the other side with message:
Invalid host key signature
06/13/2019 04:01:04 PM:[Error] SSH Client(10.202.30.163:46205) ERROR for user:N/A:Authentication failed
send reply 1 ok

[tcaduto12068]

I can duplicate it using a Python threaded Paramiko client.
You must use Paramiko version Paramiko version 2.1.2 you can install that version using PIP.

The problem occurs when the client sends one file at a time in a thread in a very fast manner.
If send files in a Python thread with no sleep in between I fails every time in the key exchange function.
Python throws this error:

File "D:\Program Files (x86)\Python35-32\lib\site-packages\paramiko\kex_group1.py", line 99, in _parse_kexdh_reply
raise SSHException('Server kex "f" is out of range')
paramiko.ssh_exception.SSHException: Server kex "f" is out of range

If I put a sleep after each thread executes, so the next one does not fire immediately it works, I can go as low as .3 seconds for the sleep any lower and the key exchange fails.

It appears something in the Key exchange method of the Sbridge server is not atomic, the connections are competing for the host key or something and it's causing issues.

Here is the sample python code where I can reproduce it:
NOTE this is using Paramiko version 2.1.2 You must use 2.1.2 to reproduce issue as the newer versions seem to work,

Code: Select all

import paramiko
import glob
import os
import time
from threading import Thread

def sendfile(filename):
    try:
        host = 'your.server.ip'
        port = 10022
        username = "testuser"
        password = "12345678"
        #print('starting thread to send file')
        transport = None
        sftp = None
        transport = paramiko.Transport((host, port))
        transport.connect(username=username, password=password)
        print('connected to host')
        sftp = paramiko.SFTPClient.from_transport(transport)
        basename = os.path.basename(filename)
        remotefilename = os.path.join('/', basename)
        sftp.put(filename, remotefilename)
        print('Uploaded file '+ filename)
    finally:
        if sftp is not None:
            sftp.close()
        if transport is not None:
            transport.close()

#Upload files 1 per connection
files = glob.glob(r'C:\Users\user1\Desktop\test_files\*')
for file in files:
    t = Thread(target=sendfile, args=(file,))
    t.start()
    #time.sleep(.5)

With the sleep set at .3 seconds the error sometimes pops up. The paramiko client gives a different error than the IBM but it seems to be the same issue:

******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:54 AM:[Error] SSH Client(10.202.56.20:55829) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55831) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Error on data reading from the connection:
An existing connection was forcibly closed by the remote host..
Socket Error Code: 10054($2746)
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55830) ERROR for user:N/A:Authentication failed

Without a doubt there is some kind of contention going on with the keyexchange and a lot of connections at one time.

[tcaduto12068]

I found a possible work around until you guys fix this:

if I put a sleep(500) in the beforeclientconnect event the issues seems to go away for smaller connection amounts.
If I send around 60 files from the python test script it will start to error out as more connections connect.

Really looks like a critical section is needed somewhere.

[tcaduto12068]

This seems to work as a temp fix:

Code: Select all

procedure TSFTPDataMod.SSHServerBeforeClientConnect(Sender: TObject;
  const SockAddr: PSockAddr; var Cancel: boolean);
var
  port,ip:string;
  randval:Integer;
begin
    Cancel:=False;
    Randomize;
    randval:= system.random(2)*100 ;

    try
        port:=intTostr(ntohs(SockAddr^.sin_port));
        ip := string(inet_ntoa(SockAddr^.sin_addr));
        push_log_msg(format('New SFTP (Before) connection from %s:%s',[ip,port]),-1,tlogtype.Connect);
    finally
        sleep(500+randval);  //<----added this
    end;
end; 

obviously this is not a idea solution as a sleep like this is a hack. You guys will have to figure out what is causing the issue with the thread contention.

[tcaduto12068]

It looks like a critical section is needed somewhere in the key exchange code.

I am using the file based keystore component, but that should not matter should it?

[tcaduto12068]

Ok, new information.
If I disable the following key exchange algorithms on the sbridge server the Python client and the Patched IBM maverick client magically start working with no sleep hacks?

keDHGroup14Sha1 and keDHGroup1Sha1

But why?

And why does it work with the above two enabled if the client does the connections one at a time or if I put a sleep to space the client send theads apart or if I add a sleep in the beforeconnect event?

Is there something in these two algorithms that is using a shared resource?

Note: the IBM system had the patch applied to update the java ssh lib and when I try the same on the unpatched production server the error persists.

Also when i have keDHGroup14Sha1 and keDHGroup1Sha1 disabled the console logging looks completely different it appears that there are critical sections in the other KEX routines but not in keDHGroup14Sha1 and keDHGroup1Sha1 .

[tcaduto12068]

Anyone from devart have comments on why this is happening?

It's super odd when the server is supposed to handle each connection in a thread that multiple fast connections cause the key exchange to fail for the older sha1 key exchange algorithms.

In my test python script if I put a .5 second delay between each thread creation it works just fine on the server no errors at all. if you guys use the same version of Paramiko (2.1.2) I used you should be able to reproduce this easily.
You can install specific version using PIP.

[ViktorV]

We continue investigating the issue taking into account the information that you have sent us. If there's any new progress we will inform you as soon as possible.

[tcaduto12068]

ok, let me know if you need me to test anything.
This is a critical issue here because they can't update the IBM software for a long time because of corporate hocus pocus LOL.

It has to be something you guys can fix since it works one file at a time or if a .5 second delay is used between sending files in the python client example for those two legacy sha1 kex algorithms. the IBM system can actually send all the files if I put a sleep(500) in the beforeclientconnect event. the ibm system does not have the ability to put a delay so the only way I can work around it right now is the sleep(500) in the beforeclientconnect event.

[tcaduto12068]

It seems when multiple connections occur very fast the keys that are getting sent back are bad.
It sends them in the SendNewKeys ok, then it dies in FCon.SyncReceivePacket and the client errors with
paramiko.ssh_exception.SSHException: Server kex "f" is out of range

I found that with the Paramiko version 2.1.2 if I disable keDHGroup14Sha1 and keDHGroup1Sha1 on the Sbridge Server It works.
This also works on the fully Patched IBM server the problem is they can't update the production servers until next year...no idea why.

[ViktorV]

We have answered you via e-mail.

[tcaduto12068]

Hi,
The issue is resolved now. My tests against the IBM sterling client relaying files to the Securebridge SFTP server completed with no errors.

Great job Devart :-) Thanks for the great support and keep up the good work.

[ViktorV]

Thank you for your interest in our product.
We are glad that the problem has been solved.
If you have any questions about our products, don't hesitate to contact us - we will do our best to help you.