SSH Client sometimes hangs on server disconnect

[cboling]

I wrote an application that runs scripted terminal sessions to remote servers. For 15 years it's supported serial and telnet, and more recently I added ssh via SBridge. Sometimes when using ssh (particularly if the ssh server is one of several products that runs on Windows) it hangs the entire program during the disconnection, requiring termination of the process.

The sbridge documentation & examples show pretty clearly the way to establish a connection. What isn't as well documented is handling disconnects. A robust program will not only do the right things in the right order under normal circumstances, but will also gracefully handle the server disconnecting at inconvenient times, not responding, etc.

Generally, my scripts will tell the server to disconnect (e.g. send "exit" to a unix shell) and then go ahead and disconnect from our end. It's then kind of a race as to which side actually does the deed -- if the server responds very quickly, it can fire a disconnect event our our end before we have had a chance to initiate a disconnect ourselves.

There are a lot of things that I don't know for sure. For example, while I would normally disconnect TScSSHShell first and then disconnect its TScSSHClient, if something happens to break my connection, might I get a Client disconnect event before/without a Shell disconnect event -- shall I attempt to call one from the other to make sure everything's cleaned up, or always treat them independently and only explicitly disconnect something if I'm initiating it? I suspect that a look at my code will show that I'm doing some stupid stuff because I didn't know any better.

Anyway, what I'm finding is that if I close things before the server does, life is always good, but if the server disconnects, sometimes it's okay, and sometimes it hangs the app.

Here's what the key parts of my Delphi code look like. (I tried to remove a few distractors while not hiding anything that could cause a problem.)

Code: Select all

Open terminal session:
              try
                frmRun.SSHCliTerm.Connect;
                try
                  frmRun.SSHShell.Connect;
                  TermSessionOpen := true;
                  TermSessionPreclose := false;
                  Log('SSH connect: (client address/port unknown)');
                except
                  on E: Exception do
                  begin
                    ErrSub(2,'SSH shell OPEN failed w/ exception: '+E.Message);
                  end;
                end;
              except
                on E: Exception do
                begin
                  ErrSub(2,'SSH connection OPEN failed w/ exception: '+E.Message);
                end;
              end;

Explanation of two variables not directly related to SBridge:

• TermSessionOpen - Will be set to false when we intentionally close the connection, or otherwise are made aware that it's gone, and have completed the appropriate cleanup
• TermSessionPreclose - We can set to true before telling a server to disconnect, so that if we get a disconnect event, we know if we asked for it or not.

Those two things really don't affect anything substantial; they just control whether we consider things (such as a disconnect event) to be errors or expected events.

Code: Select all

(Initiating close from our end:)
       tmSSH: begin
          if TermSessionOpen then
          begin
            TermSessionOpen := false;
          end else
          begin
            log('(SSH session already closed)');
          end;
          If frmRun.SSHShell.Connected then
          begin
            log('(Calling SSHShell.Disconnect)');
            frmRun.SSHShell.Disconnect;
          end else
            log('(SSHShell.Connected is already false)');
          If frmRun.SSHCliTerm.Connected then
          begin
            log('(Calling SSHCliTerm.Disconnect)');
            frmRun.SSHCliTerm.Disconnect;
          end else
            log('(SSHCliTerm.Connected is already false)');
        end;

Code: Select all

procedure TfrmRun.SSHShellDisconnect(Sender: TObject);
  Log('Event: SSHShell Disconnected');
  Log('debug a');
  TermBuffAdd('{DISCONNECTED}');
  Log('debug b');
  If TermSessionOpen then Log('(Server closed terminal session)');
  Log('debug c');
  If frmRun.SSHCliTerm.Connected then 
  begin
    Log('debug d');
    log('(Calling SSHCliTerm.Disconnect from SSHShellDisconnect event)');
    Log('debug e');
    frmRun.SSHCliTerm.Disconnect;
    Log('debug f');
  end else
    log('(SSHShell.Connected is already false in SSHShellDisconnect event)');

Code: Select all

procedure TfrmRun.SSHCliTermAfterDisconnect(Sender: TObject);
begin
    Log('Event: SSHCliTerm Disconnected -- just sayin''.');
end;

Here are log samples from runs showing these conditions:

• We completed disconnect first
• We saw server disconnect before we finished disconnecting
• Server disconnected and we hung

We completed disconnect first (okay):

#30 terminal open
Event: SSH Connected
SSH connect: (client address/port unknown)
#31 sleep 1
#32 sendln exit
#33 #sleep 1
#34 terminal close
(Calling SSHShell.Disconnect)
Event: SSHShell Disconnected
debug a
debug b
debug c
debug d
(Calling SSHCliTerm.Disconnect from SSHShellDisconnect event)
debug e
Event: SSHShell Disconnected
debug a
debug b
debug c
(SSHShell.Connected is already false in SSHShellDisconnect event)
Event: SSHCliTerm Disconnected -- just sayin'.
(SSHCliTerm.Connected is already false)
Job finished. Elapsed Time: 00:01

I do see that the SSHShellDisconnect is invoked twice...perhaps automatically called from SSHCliTerm.Disconnect event, or triggered by a server-side disconnect?

The server disconnected (still okay):

#32 sendln exit
Event: SSHShell Disconnected
debug a
debug b
Server closed terminal session
debug c
debug d
(Calling SSHCliTerm.Disconnect from SSHShellDisconnect event)
debug e
Event: SSHShell Disconnected
debug a
debug b
debug c
(SSHShell.Connected is already false in SSHShellDisconnect event)
Event: SSHCliTerm Disconnected -- just sayin'.
#33 #sleep 1
#34 terminal close
(SSH session already closed)
(SSHShell.Connected is already false)
(SSHCliTerm.Connected is already false)
Job finished. Elapsed Time: 00:01

The server disconnected (hung the app):

#606 SendLn OFF
#607 Return
#611 Terminal Close
Event: SSHShell Disconnected
debug a
debug b
debug c
debug d
debug e
Event: SSHShell Disconnected
debug a
debug b
debug c

d,e,f are only executed if SSHCliTerm.Connected, which is why they aren't executed the 2nd time SSHShell.disconnect is called.

Another case where the server disconnected & hung the app:

#559 Terminal Close
(Calling SSHShell.Disconnect)
Event: SSHShell Disconnected
debug a
debug b
debug c
debug d
(Calling SSHCliTerm.Disconnect from SSHShellDisconnect event)
debug e
Event: SSHShell Disconnected
debug a
debug b
debug c
(SSHShell.Connected is already false in SSHShellDisconnect event)
Event: SSHCliTerm Disconnected -- just sayin'.

I don't know exactly *what* is hanging after that. I've been unable to reproduce the problem in a development environment (so I've just been sending out special builds w/ debugging messages), but I just got info that may allow me to build a server that will allow me to reproduce the problem while running in the IDE (not that debugging multiple threads & events would be pretty anyway).

Any info & advice you can provide would be greatly appreciated!

[ViktorV]

We have conducted multiple tests of these problem, but couldn't reproduce it: the application never hanged. To investigate the issue, please compose a complete sample demonstrating the issue and send it to us.

[cboling]

Thanks, Viktor; I'll work on that. It's a complex program that's dependent on several other 3rd party controls, so if it takes *too* much work to strip it down, I may provide you a complete dev box including a server that you can to connect to and play on, if that's okay.

Meanwhile...so you didn't see anything too bad in my code? Anything that I'm doing out of order, or shouldn't do, or definitely should do just for good practice (even though it may not be causing the crash)?

[cboling]

After spending 30+ hours building a clean dev box for testing (my existing one had grown organically over 15 years, and wasn't well-documented), I believe I found the trouble spot:

(Calling SSHCliTerm.Disconnect from SSHShellDisconnect event)

Do not call a TScSSHClient.Disconnect handler from within its TScSSHShell.OnDisconnect event! Though it won't fire automatically after a shell disconnect, you need to wait until after the shell disconnect event is complete, then call it.

Does that sound right, Viktor?

[ViktorV]

Yes, you are right. You shouldn't call the TScSSHClient.Disconnect method from the TScSSHShell.OnDisconnect event handler.