TScSSHClient: Access violation error on reconnect

[cboling]

My SSH client tries to establish & maintain a connection w/ the server, where "connection" means all 3 of these:
- The SSH connection itself (TScSSHClient)
- a terminal session (TScSSHShell)
- a port-forwarding tunnel (TScSSHChannel)

When it detects that it has lost at least one of these, it ensures that everything else is disconnected, waits, then tries to connect again.

I must be doing something wrong because often, following the disconnection, the next TScSSHClient connect fails w/ a memory access violation error. The 2nd attempt to reconnect is usually successful. Do you see anything wrong with my logic below?

Here's the logic for connecting to the server:

Code: Select all

C3Connect:
	Set HostName, Port, HostKeyName, User, Password, Authentication
	Try
		SSHClient.Connect
		Try
			SSHShell.Connect
			Send a shell command, sleep for a couple of seconds
			Try
				SSHChannel.Connect
			Except log channel exception
		Except log shell exception
	Except log client exception

I don't really do anything at the time of disconnection:
OnDisconnect event logics:

• Shell: Disable telemetry timer & log event
• Channel: Log event
• Client: Log event

Every 5 seconds via a timer:

Code: Select all

	Disable timer to prevent reentry
	Do unrelated things
	If at least a minute has passed since the last connection attempt
		If either Shell or Channel isn't connected
			Log it & disconnect the other
			Disconnect Client
		If Client isn't connected
			C3Connect (see function above)
	Do more unrelated things
	Re-enable timer

(Since the SSH components don't complain if I call .disconnect for something that's already disconnected, I don't bother to check connection status before I call it.)

This is a typical sequence of events in my debugging log:

2020-06-22 17:05:00.385 - AppLog:C3 shell disconnected
2020-06-22 17:05:00.385 - AppLog:C3 SSH disconnected
2020-06-22 17:05:02.803 - Timer1 event
2020-06-22 17:05:02.803 - Timer1 - SSH shell is disconnected; ensuring that the TN tunnel is too.
2020-06-22 17:05:02.804 - AppLog:C3TN channel disconnected
2020-06-22 17:05:02.805 - Timer1 - Disconnecting SSH connection
2020-06-22 17:05:03.006 - Timer1 before C3Connect
2020-06-22 17:06:36.233 - AppLog:C3 connection failed w/ exception: Access violation at address 00000040. Read of address 00000040
2020-06-22 17:06:36.234 - Timer1 after C3Connect

Running it within the IDE, after it stops w/ the error, the Delphi call stack shows:

Code: Select all

	Timer
		C3Connect
			TScSSHClient.Connect
				TScSSHClient.SetConnected

I set a breakpoint and stepped in, and here's what I narrowed it down to:

Code: Select all

SetConnected
	TScSSHClient.DoConnect
		FreeAndNil(FConnection)     -- I set a breakpoint here and traced in w/ F7
			TSsh2Connection.Destroy
				TSsh2Connection.Close;
					FStream.Vio.Close    -- this is what triggers the Access Violation

PS: Using SecureBridge 9.2.4 w/ src on Delphi 7

[ViktorV]

Unfortunately, we haven't been able to reproduce the issue.
Please create a sample project demonstrating the incorrect behavior and send it to us through the contact form (https://devart.com/company/contactform.html).

[cboling]

This was a very surprising devlopment:
The problem disappeared after switching from Delphi's default memory manager to FastMM4.

Unneccessary details:
I've been testing stability on about 20 machines in various locations. One particular machine, for unknown reasons, gets its network connections dropped frequently -- 10-30x/day (Interestingly, two other instances running inside a VM on that same host are rock-solid) -- so it's great for exposing problems w/ the reconnection without having to manually break connections. This A/V error shows up roughly 90% (but not 100%) of the time after those reconnects.

Yesterday, while awaiting ideas on this issue, I decided to tackle a low-priority project: installing FastMM4 for Delphi and seeing what it did for both the IDE and my program in terms of performance and memory fragmentation. I pushed out a new version of my program yesterday afternoon. This morning, I was shocked to find that, since sending out the new version (which had no change other than being compiled with the new MM) that the "problem machine" showed 11 new disconnect/reconnects -- without a single error. I tested on another machine, intentionally breaking the connection 25 times over a couple of hours, and not a single error -- whereas before, as I said, 90% of the time the error would appear.

The only explanation I can think of is some sort of race condition and the new MM affected the timing in a helpful way; either that or there was a serious bug in Delphi 7's MM...but such a thing should've been really obvious in other applications too. No need to pursue this any more -- the problem's gone -- but it's sure interesting!

[ViktorV]

We are glad to see you have found a solution.
Feel free to contact us if you have any further questions about our products.