UPDATED Please read: Host key signature failure error
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: Host key signature failure error message
Hi,
We updated the IBM system and we are still getting the error:
( I put some logging in the functions) It gets to the SyncKeyExchange and the other side rejects the
signature Sbridge is sending)
Is there a way to capture what it's sending to verify it's not changing the host key signature?
send init ok
process init ok
KeyExchangeAlgorithm 1 ok
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Connection was closed by the other side with message:
Invalid host key signature
06/13/2019 04:01:04 PM:[Error] SSH Client(10.202.30.163:46205) ERROR for user:N/A:Authentication failed
send reply 1 ok
We updated the IBM system and we are still getting the error:
( I put some logging in the functions) It gets to the SyncKeyExchange and the other side rejects the
signature Sbridge is sending)
Is there a way to capture what it's sending to verify it's not changing the host key signature?
send init ok
process init ok
KeyExchangeAlgorithm 1 ok
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Connection was closed by the other side with message:
Invalid host key signature
06/13/2019 04:01:04 PM:[Error] SSH Client(10.202.30.163:46205) ERROR for user:N/A:Authentication failed
send reply 1 ok
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: Host key signature failure error message
I can duplicate it using a Python threaded Paramiko client.
You must use Paramiko version Paramiko version 2.1.2 you can install that version using PIP.
The problem occurs when the client sends one file at a time in a thread in a very fast manner.
If send files in a Python thread with no sleep in between I fails every time in the key exchange function.
Python throws this error:
File "D:\Program Files (x86)\Python35-32\lib\site-packages\paramiko\kex_group1.py", line 99, in _parse_kexdh_reply
raise SSHException('Server kex "f" is out of range')
paramiko.ssh_exception.SSHException: Server kex "f" is out of range
If I put a sleep after each thread executes, so the next one does not fire immediately it works, I can go as low as .3 seconds for the sleep any lower and the key exchange fails.
It appears something in the Key exchange method of the Sbridge server is not atomic, the connections are competing for the host key or something and it's causing issues.
Here is the sample python code where I can reproduce it:
NOTE this is using Paramiko version 2.1.2 You must use 2.1.2 to reproduce issue as the newer versions seem to work,
With the sleep set at .3 seconds the error sometimes pops up. The paramiko client gives a different error than the IBM but it seems to be the same issue:
******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:54 AM:[Error] SSH Client(10.202.56.20:55829) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55831) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Error on data reading from the connection:
An existing connection was forcibly closed by the remote host..
Socket Error Code: 10054($2746)
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55830) ERROR for user:N/A:Authentication failed
Without a doubt there is some kind of contention going on with the keyexchange and a lot of connections at one time.
You must use Paramiko version Paramiko version 2.1.2 you can install that version using PIP.
The problem occurs when the client sends one file at a time in a thread in a very fast manner.
If send files in a Python thread with no sleep in between I fails every time in the key exchange function.
Python throws this error:
File "D:\Program Files (x86)\Python35-32\lib\site-packages\paramiko\kex_group1.py", line 99, in _parse_kexdh_reply
raise SSHException('Server kex "f" is out of range')
paramiko.ssh_exception.SSHException: Server kex "f" is out of range
If I put a sleep after each thread executes, so the next one does not fire immediately it works, I can go as low as .3 seconds for the sleep any lower and the key exchange fails.
It appears something in the Key exchange method of the Sbridge server is not atomic, the connections are competing for the host key or something and it's causing issues.
Here is the sample python code where I can reproduce it:
NOTE this is using Paramiko version 2.1.2 You must use 2.1.2 to reproduce issue as the newer versions seem to work,
Code: Select all
import paramiko
import glob
import os
import time
from threading import Thread
def sendfile(filename):
try:
host = 'your.server.ip'
port = 10022
username = "testuser"
password = "12345678"
#print('starting thread to send file')
transport = None
sftp = None
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
print('connected to host')
sftp = paramiko.SFTPClient.from_transport(transport)
basename = os.path.basename(filename)
remotefilename = os.path.join('/', basename)
sftp.put(filename, remotefilename)
print('Uploaded file '+ filename)
finally:
if sftp is not None:
sftp.close()
if transport is not None:
transport.close()
#Upload files 1 per connection
files = glob.glob(r'C:\Users\user1\Desktop\test_files\*')
for file in files:
t = Thread(target=sendfile, args=(file,))
t.start()
#time.sleep(.5)
******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:54 AM:[Error] SSH Client(10.202.56.20:55829) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Socket closed. Cannot receive data
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55831) ERROR for user:N/A:Authentication failed
send reply 1 ok
send new keys ok
******************************** Exception in SyncKeyExchange:Error on data reading from the connection:
An existing connection was forcibly closed by the remote host..
Socket Error Code: 10054($2746)
06/14/2019 11:49:55 AM:[Error] SSH Client(10.202.56.20:55830) ERROR for user:N/A:Authentication failed
Without a doubt there is some kind of contention going on with the keyexchange and a lot of connections at one time.
Last edited by tcaduto12068 on Mon 17 Jun 2019 21:27, edited 7 times in total.
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: Host key signature failure error message
I found a possible work around until you guys fix this:
if I put a sleep(500) in the beforeclientconnect event the issues seems to go away for smaller connection amounts.
If I send around 60 files from the python test script it will start to error out as more connections connect.
Really looks like a critical section is needed somewhere.
if I put a sleep(500) in the beforeclientconnect event the issues seems to go away for smaller connection amounts.
If I send around 60 files from the python test script it will start to error out as more connections connect.
Really looks like a critical section is needed somewhere.
Last edited by tcaduto12068 on Fri 14 Jun 2019 21:50, edited 2 times in total.
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: Host key signature failure error message
This seems to work as a temp fix:
obviously this is not a idea solution as a sleep like this is a hack. You guys will have to figure out what is causing the issue with the thread contention.
Code: Select all
procedure TSFTPDataMod.SSHServerBeforeClientConnect(Sender: TObject;
const SockAddr: PSockAddr; var Cancel: boolean);
var
port,ip:string;
randval:Integer;
begin
Cancel:=False;
Randomize;
randval:= system.random(2)*100 ;
try
port:=intTostr(ntohs(SockAddr^.sin_port));
ip := string(inet_ntoa(SockAddr^.sin_addr));
push_log_msg(format('New SFTP (Before) connection from %s:%s',[ip,port]),-1,tlogtype.Connect);
finally
sleep(500+randval); //<----added this
end;
end;
Last edited by tcaduto12068 on Fri 14 Jun 2019 18:31, edited 1 time in total.
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: Host key signature failure error message
It looks like a critical section is needed somewhere in the key exchange code.
I am using the file based keystore component, but that should not matter should it?
I am using the file based keystore component, but that should not matter should it?
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Please Read Updated: Host key signature failure error message
Ok, new information.
If I disable the following key exchange algorithms on the sbridge server the Python client and the Patched IBM maverick client magically start working with no sleep hacks?
keDHGroup14Sha1 and keDHGroup1Sha1
But why?
And why does it work with the above two enabled if the client does the connections one at a time or if I put a sleep to space the client send theads apart or if I add a sleep in the beforeconnect event?
Is there something in these two algorithms that is using a shared resource?
Note: the IBM system had the patch applied to update the java ssh lib and when I try the same on the unpatched production server the error persists.
Also when i have keDHGroup14Sha1 and keDHGroup1Sha1 disabled the console logging looks completely different it appears that there are critical sections in the other KEX routines but not in keDHGroup14Sha1 and keDHGroup1Sha1 .
If I disable the following key exchange algorithms on the sbridge server the Python client and the Patched IBM maverick client magically start working with no sleep hacks?
keDHGroup14Sha1 and keDHGroup1Sha1
But why?
And why does it work with the above two enabled if the client does the connections one at a time or if I put a sleep to space the client send theads apart or if I add a sleep in the beforeconnect event?
Is there something in these two algorithms that is using a shared resource?
Note: the IBM system had the patch applied to update the java ssh lib and when I try the same on the unpatched production server the error persists.
Also when i have keDHGroup14Sha1 and keDHGroup1Sha1 disabled the console logging looks completely different it appears that there are critical sections in the other KEX routines but not in keDHGroup14Sha1 and keDHGroup1Sha1 .
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: UPDATED Please read: Host key signature failure error
Anyone from devart have comments on why this is happening?
It's super odd when the server is supposed to handle each connection in a thread that multiple fast connections cause the key exchange to fail for the older sha1 key exchange algorithms.
In my test python script if I put a .5 second delay between each thread creation it works just fine on the server no errors at all. if you guys use the same version of Paramiko (2.1.2) I used you should be able to reproduce this easily.
You can install specific version using PIP.
It's super odd when the server is supposed to handle each connection in a thread that multiple fast connections cause the key exchange to fail for the older sha1 key exchange algorithms.
In my test python script if I put a .5 second delay between each thread creation it works just fine on the server no errors at all. if you guys use the same version of Paramiko (2.1.2) I used you should be able to reproduce this easily.
You can install specific version using PIP.
Re: UPDATED Please read: Host key signature failure error
We continue investigating the issue taking into account the information that you have sent us. If there's any new progress we will inform you as soon as possible.
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: UPDATED Please read: Host key signature failure error
ok, let me know if you need me to test anything.
This is a critical issue here because they can't update the IBM software for a long time because of corporate hocus pocus LOL.
It has to be something you guys can fix since it works one file at a time or if a .5 second delay is used between sending files in the python client example for those two legacy sha1 kex algorithms. the IBM system can actually send all the files if I put a sleep(500) in the beforeclientconnect event. the ibm system does not have the ability to put a delay so the only way I can work around it right now is the sleep(500) in the beforeclientconnect event.
This is a critical issue here because they can't update the IBM software for a long time because of corporate hocus pocus LOL.
It has to be something you guys can fix since it works one file at a time or if a .5 second delay is used between sending files in the python client example for those two legacy sha1 kex algorithms. the IBM system can actually send all the files if I put a sleep(500) in the beforeclientconnect event. the ibm system does not have the ability to put a delay so the only way I can work around it right now is the sleep(500) in the beforeclientconnect event.
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: UPDATED Please read: Host key signature failure error
It seems when multiple connections occur very fast the keys that are getting sent back are bad.
It sends them in the SendNewKeys ok, then it dies in FCon.SyncReceivePacket and the client errors with
paramiko.ssh_exception.SSHException: Server kex "f" is out of range
I found that with the Paramiko version 2.1.2 if I disable keDHGroup14Sha1 and keDHGroup1Sha1 on the Sbridge Server It works.
This also works on the fully Patched IBM server the problem is they can't update the production servers until next year...no idea why.
It sends them in the SendNewKeys ok, then it dies in FCon.SyncReceivePacket and the client errors with
paramiko.ssh_exception.SSHException: Server kex "f" is out of range
I found that with the Paramiko version 2.1.2 if I disable keDHGroup14Sha1 and keDHGroup1Sha1 on the Sbridge Server It works.
This also works on the fully Patched IBM server the problem is they can't update the production servers until next year...no idea why.
Re: UPDATED Please read: Host key signature failure error
We have answered you via e-mail.
-
- Posts: 132
- Joined: Wed 17 Aug 2016 05:57
Re: UPDATED Please read: Host key signature failure error
Hi,
The issue is resolved now. My tests against the IBM sterling client relaying files to the Securebridge SFTP server completed with no errors.
Great job Devart :-) Thanks for the great support and keep up the good work.
The issue is resolved now. My tests against the IBM sterling client relaying files to the Securebridge SFTP server completed with no errors.
Great job Devart :-) Thanks for the great support and keep up the good work.
Re: UPDATED Please read: Host key signature failure error
Thank you for your interest in our product.
We are glad that the problem has been solved.
If you have any questions about our products, don't hesitate to contact us - we will do our best to help you.
We are glad that the problem has been solved.
If you have any questions about our products, don't hesitate to contact us - we will do our best to help you.