UPDATED Please read: Host key signature failure error

Discussion of open issues, suggestions and bugs regarding network security and data protection solution - SecureBridge
tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: Host key signature failure error message

Post by tcaduto12068 » Thu 13 Jun 2019 21:14

Hi,
We updated the IBM system and we are still getting the error:
( I put some logging in the functions) It gets to the SyncKeyExchange and the other side rejects the
signature Sbridge is sending)

Is there a way to capture what it's sending to verify it's not changing the host key signature?

send init ok
process init ok
KeyExchangeAlgorithm 1 ok
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Connection was closed by the other side with message:
Invalid host key signature
06/13/2019 04:01:04 PM:[Error] SSH Client(10.202.30.163:46205) ERROR for user:N/A:Authentication failed
send reply 1 ok

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: Host key signature failure error message

Post by tcaduto12068 » Fri 14 Jun 2019 16:34

I can duplicate it using a Python threaded Paramiko client.
You must use Paramiko version Paramiko version 2.1.2 you can install that version using PIP.

The problem occurs when the client sends one file at a time in a thread in a very fast manner.
If send files in a Python thread with no sleep in between I fails every time in the key exchange function.
Python throws this error:

File "D:\Program Files (x86)\Python35-32\lib\site-packages\paramiko\kex_group1.py", line 99, in _parse_kexdh_reply
raise SSHException('Server kex "f" is out of range')
paramiko.ssh_exception.SSHException: Server kex "f" is out of range

If I put a sleep after each thread executes, so the next one does not fire immediately it works, I can go as low as .3 seconds for the sleep any lower and the key exchange fails.

It appears something in the Key exchange method of the Sbridge server is not atomic, the connections are competing for the host key or something and it's causing issues.

Here is the sample python code where I can reproduce it:
NOTE this is using Paramiko version 2.1.2 You must use 2.1.2 to reproduce issue as the newer versions seem to work,

Code: Select all


import paramiko
import glob
import os
import time
from threading import Thread

def sendfile(filename):
    try:
        host = 'your.server.ip'
        port = 10022
        username = "testuser"
        password = "12345678"
        #print('starting thread to send file')
        transport = None
        sftp = None
        transport = paramiko.Transport((host, port))
        transport.connect(username=username, password=password)
        print('connected to host')
        sftp = paramiko.SFTPClient.from_transport(transport)
        basename = os.path.basename(filename)
        remotefilename = os.path.join('/', basename)
        sftp.put(filename, remotefilename)
        print('Uploaded file '+ filename)
    finally:
        if sftp is not None:
            sftp.close()
        if transport is not None:
            transport.close()

#Upload files 1 per connection
files = glob.glob(r'C:\Users\user1\Desktop\test_files\*')
for file in files:
    t = Thread(target=sendfile, args=(file,))
    t.start()
    #time.sleep(.5)

With the sleep set at .3 seconds the error sometimes pops up. The paramiko client gives a different error than the IBM but it seems to be the same issue:

******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:54 AM:[Error] SSH Client(10.202.56.20:55829) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55831) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Error on data reading from the connection:
An existing connection was forcibly closed by the remote host..
Socket Error Code: 10054($2746)
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55830) ERROR for user:N/A:Authentication failed

Without a doubt there is some kind of contention going on with the keyexchange and a lot of connections at one time.
Last edited by tcaduto12068 on Mon 17 Jun 2019 21:27, edited 7 times in total.

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: Host key signature failure error message

Post by tcaduto12068 » Fri 14 Jun 2019 17:12

I found a possible work around until you guys fix this:

if I put a sleep(500) in the beforeclientconnect event the issues seems to go away for smaller connection amounts.
If I send around 60 files from the python test script it will start to error out as more connections connect.

Really looks like a critical section is needed somewhere.
Last edited by tcaduto12068 on Fri 14 Jun 2019 21:50, edited 2 times in total.

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: Host key signature failure error message

Post by tcaduto12068 » Fri 14 Jun 2019 17:19

This seems to work as a temp fix:

Code: Select all

procedure TSFTPDataMod.SSHServerBeforeClientConnect(Sender: TObject;
  const SockAddr: PSockAddr; var Cancel: boolean);
var
  port,ip:string;
  randval:Integer;
begin
    Cancel:=False;
    Randomize;
    randval:= system.random(2)*100 ;

    try
        port:=intTostr(ntohs(SockAddr^.sin_port));
        ip := string(inet_ntoa(SockAddr^.sin_addr));
        push_log_msg(format('New SFTP (Before) connection from %s:%s',[ip,port]),-1,tlogtype.Connect);
    finally
        sleep(500+randval);  //<----added this
    end;
end; 
obviously this is not a idea solution as a sleep like this is a hack. You guys will have to figure out what is causing the issue with the thread contention.
Last edited by tcaduto12068 on Fri 14 Jun 2019 18:31, edited 1 time in total.

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: Host key signature failure error message

Post by tcaduto12068 » Fri 14 Jun 2019 18:30

It looks like a critical section is needed somewhere in the key exchange code.

I am using the file based keystore component, but that should not matter should it?

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Please Read Updated: Host key signature failure error message

Post by tcaduto12068 » Mon 17 Jun 2019 18:45

Ok, new information.
If I disable the following key exchange algorithms on the sbridge server the Python client and the Patched IBM maverick client magically start working with no sleep hacks?

keDHGroup14Sha1 and keDHGroup1Sha1

But why?

And why does it work with the above two enabled if the client does the connections one at a time or if I put a sleep to space the client send theads apart or if I add a sleep in the beforeconnect event?

Is there something in these two algorithms that is using a shared resource?

Note: the IBM system had the patch applied to update the java ssh lib and when I try the same on the unpatched production server the error persists.

Also when i have keDHGroup14Sha1 and keDHGroup1Sha1 disabled the console logging looks completely different it appears that there are critical sections in the other KEX routines but not in keDHGroup14Sha1 and keDHGroup1Sha1 .

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: UPDATED Please read: Host key signature failure error

Post by tcaduto12068 » Tue 18 Jun 2019 13:36

Anyone from devart have comments on why this is happening?

It's super odd when the server is supposed to handle each connection in a thread that multiple fast connections cause the key exchange to fail for the older sha1 key exchange algorithms.

In my test python script if I put a .5 second delay between each thread creation it works just fine on the server no errors at all. if you guys use the same version of Paramiko (2.1.2) I used you should be able to reproduce this easily.
You can install specific version using PIP.

ViktorV
Devart Team
Posts: 2732
Joined: Wed 30 Jul 2014 07:16

Re: UPDATED Please read: Host key signature failure error

Post by ViktorV » Tue 18 Jun 2019 14:50

We continue investigating the issue taking into account the information that you have sent us. If there's any new progress we will inform you as soon as possible.

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: UPDATED Please read: Host key signature failure error

Post by tcaduto12068 » Tue 18 Jun 2019 15:36

ok, let me know if you need me to test anything.
This is a critical issue here because they can't update the IBM software for a long time because of corporate hocus pocus LOL.

It has to be something you guys can fix since it works one file at a time or if a .5 second delay is used between sending files in the python client example for those two legacy sha1 kex algorithms. the IBM system can actually send all the files if I put a sleep(500) in the beforeclientconnect event. the ibm system does not have the ability to put a delay so the only way I can work around it right now is the sleep(500) in the beforeclientconnect event.

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: UPDATED Please read: Host key signature failure error

Post by tcaduto12068 » Thu 20 Jun 2019 20:18

It seems when multiple connections occur very fast the keys that are getting sent back are bad.
It sends them in the SendNewKeys ok, then it dies in FCon.SyncReceivePacket and the client errors with
paramiko.ssh_exception.SSHException: Server kex "f" is out of range

I found that with the Paramiko version 2.1.2 if I disable keDHGroup14Sha1 and keDHGroup1Sha1 on the Sbridge Server It works.
This also works on the fully Patched IBM server the problem is they can't update the production servers until next year...no idea why.

ViktorV
Devart Team
Posts: 2732
Joined: Wed 30 Jul 2014 07:16

Re: UPDATED Please read: Host key signature failure error

Post by ViktorV » Fri 21 Jun 2019 14:44

We have answered you via e-mail.

tcaduto12068
Posts: 129
Joined: Wed 17 Aug 2016 05:57

Re: UPDATED Please read: Host key signature failure error

Post by tcaduto12068 » Tue 25 Jun 2019 16:02

Hi,
The issue is resolved now. My tests against the IBM sterling client relaying files to the Securebridge SFTP server completed with no errors.

Great job Devart :-) Thanks for the great support and keep up the good work.

ViktorV
Devart Team
Posts: 2732
Joined: Wed 30 Jul 2014 07:16

Re: UPDATED Please read: Host key signature failure error

Post by ViktorV » Wed 26 Jun 2019 13:37

Thank you for your interest in our product.
We are glad that the problem has been solved.
If you have any questions about our products, don't hesitate to contact us - we will do our best to help you.

Post Reply