SFTP - UploadAppend ?

gw · Post by gw » Fri 05 Nov 2010 13:12

i have a present application written with Borland C++ Builder 5 (BCB5) that uses the NMFTP component of BCB5. Now this application needs to be changed from FTP to SFTP. I intend to use SecureBridge to do this.

I have one question about that:
In the present application i (among others) used the UploadAppend function of the NMFTP component and as far as i have seen there seems to be no such function in SecureBridge.
Can this (easily) be done with other functions (like maybe TScSFTPClient.WriteFile)?

thanks
/gw

Dimon · Post by **Dimon** » Mon 08 Nov 2010 15:11

You can implement this functionality by performing the following steps:
1. Establish SFTP connection using the TScSFTPClient.Initialize method.
2. Open a remote file by the OpenFile method with the Modes parameter equaled [foAppend, foCreate].
3. Write data to the file using the WriteFile method.
You can find more detailed information about these methods in the SecureBridge help.

gw · Post by gw » Fri 18 Mar 2011 17:39

So, license is now bought.
Most things i need are working now (like connecting, read dir, download file, remove file) but with this UploadAppend replacement i'm still having trouble.
Here is my current code of that function:

Code: Select all

bool __fastcall TSftpForm::UploadProgress(char *localfile)
{
  // replacement for UploadAppend of NTFTP component of BCB5
  Scclrclasses::TBytes Handle;
  Scsftputils::TScSFTPFileOpenModes FOM;
  Scsftputils::TScSFTPFileAttributes* FA = new(Scsftputils::TScSFTPFileAttributes);
  AnsiString Directory;
  int i,handle,bytes,blocks,rest;
  long filelen;
#define BUFFER_SIZE 32768
  byte Puffer[BUFFER_SIZE];

  FtpError = false;
  handle = open(localfile, O_RDONLY|O_BINARY);
  if (handle 0)
  {
  ConnectToFtp();
  //the upload directory is one level up (to the download directory)
  if (MaschPar.lov[MaschPar.LovSelection].Directory.Pos("\") > 0)
    Directory = MaschPar.lov[MaschPar.LovSelection].Directory.SubString(1,MaschPar.lov[MaschPar.LovSelection].Directory.LastDelimiter("\"));
  else
    Directory = MaschPar.lov[MaschPar.LovSelection].Directory.SubString(1,MaschPar.lov[MaschPar.LovSelection].Directory.LastDelimiter("/"));
  try
  {
    try
    {
      //NMFTP->UploadAppend(localfile,MaschPar.lov[MaschPar.LovSelection].Output); //this was real easy
      FOM Permissions = Scsftputils::TScSFTPFilePermissions() ValidAttributes > aSize >> aAllocationSize >> aOwnerGroup
                          >> aAccessTime >> aCreateTime >> aModifyTime >> aChangeAttrTime
                          >> aSubsecondTimes >> aAcl >> aAttrs >> aTextHint >> aMimeType
                          >> aLinkCount >> aUntranslatedName >> aExtended;
      Handle = ScSFTPClient->OpenFile(Directory+MaschPar.lov[MaschPar.LovSelection].Output,FOM,FA);
      if (FtpError)
      {
        close(handle);
        DisconnectFromFtp();
        return FtpError;
      }
      blocks = filelen / BUFFER_SIZE;
      rest = filelen % BUFFER_SIZE;
      for (i=0;iWriteFile(Handle,0,(void *)&Puffer[0],bytes);
      }
      if (rest!=0)
      {
        if ((bytes = read(handle, Puffer, rest)) != rest)
        {
          close(handle);
          DisconnectFromFtp();
          return true;
        }
        ScSFTPClient->WriteFile(Handle,0,(void *)&Puffer[0],bytes);
      }
      // at the latest (try to) do a "chmod 664" here
      FA->Permissions = Scsftputils::TScSFTPFilePermissions() ValidAttributes > aSize >> aAllocationSize >> aOwnerGroup
                          >> aAccessTime >> aCreateTime >> aModifyTime >> aChangeAttrTime
                          >> aSubsecondTimes >> aAcl >> aAttrs >> aTextHint >> aMimeType
                          >> aLinkCount >> aUntranslatedName >> aExtended;
      ScSFTPClient->SetAttributesByHandle(Handle,FA);
      ScSFTPClient->CloseHandle(Handle);
    }
    __except( EXCEPTION_EXECUTE_HANDLER )
    {
      FtpError = true;
    }
  }
  catch ( ... )
  {
    FtpError = true;
  }
  DisconnectFromFtp();
  }
  close(handle);
  return FtpError;
}

The (remote)file is opened/created but the 'WriteFile' commands fail with "Permission denied", error code 3 on operation opWritingFile.
I think it has to do with the TScSFTPFileAttributes (ace-mask flags ?).
The customer told me already to specify "chmod 664" on the file, that's what i've set in FA->Permissions.
What Attributes need to be set and how (what values to use to initialize)?
A short code snippet (preferable in C++) would be nice.
Currently i'm (locally) testing using "freeFTPd" as SFTP server under Windows.
Customer later will have a SFTP server running under Linux.

many thanks in advance
best regards
/gw

gw · Post by gw » Mon 21 Mar 2011 12:30

Still need help on this!
When i use

Code: Select all

UploadFile(localfile,Directory+MaschPar.lov[MaschPar.LovSelection].Output);

instead then it works (but the handle on the local file has to be closed for that). But in some cases the remote file can contain old data (from a previous upload) that shouldn't get lost. I could check for presence and size of remote file and in case download it, append locally and then upload the append file again, but i'd like to do it with "UploadAppend" as before. The data in this file contains a production progress report and is read and analysed, then deleted by an other application. The interval when this is done can be higher than (new) progress reports have to be uploaded.
Is this combination of OpenFile (foCreate, foAppend) with WriteFile ever tested/used by somebody?
Are these

Code: Select all

1.      FOM Permissions = Scsftputils::TScSFTPFilePermissions() ValidAttributes > aSize >> aAllocationSize >> aOwnerGroup 
                          >> aAccessTime >> aCreateTime >> aModifyTime >> aChangeAttrTime 
                          >> aSubsecondTimes >> aAcl >> aAttrs >> aTextHint >> aMimeType 
                          >> aLinkCount >> aUntranslatedName >> aExtended;

instructions ok? I think 2 and 3 are but on 1 i'm not sure, but have tried also other notation (like FOM = FOM << foAppend << foCreate;) without success (when i want to inspect the value it always shows empty brackets "{ }".
Hope to get some help or useful ideas soon.
I need help urgently!

best regards
/gw

gw · Post by gw » Wed 23 Mar 2011 11:31

A)
I changed to Cerberus FTP Server (still under Windows) for testing, and now the "Permission denied" error is gone

BUT the append is still not working, the file seems to be created/truncated - i only see the new content, the old one is gone

I also tried:

Code: Select all

      Handle = ScSFTPClient->OpenFile(Directory+MaschPar.lov[MaschPar.LovSelection].Output,Scsftputils::TScSFTPFileOpenModes() Permissions = Scsftputils::TScSFTPFilePermissions() ValidAttributes << aPermissions;

As far as i have noticed on debugging, statement 2 seems to be unnecessary, as statement 1 sets 'aPermissions' in 'ValidAttributes' by itself - is that right?

best regards
/gw

gw · Post by gw » Thu 24 Mar 2011 14:05

Now tested with JSCAPE MFT Server as a third try, with the following result:
"error opening file", error code 4 on operation opOpeningFile
Log of FTP Server:

2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - SSH_MSG_NEWKEYS id=2 - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - - - SSH_MSG_NEWKEYS id=2 - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - SSH_MSG_SERVICE_REQUEST "id=3; service_name=ssh-userauth" - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - - - SSH_MSG_SERVICE_ACCEPT "id=3; service_name=ssh-userauth" - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - SSH_MSG_USERAUTH_REQUEST "id=4; username=MyUser; service=ssh-connection; method=none" - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - - - SSH_MSG_USERAUTH_FAILURE "id=4; authentications=password; partial_success=false" - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - SSH_MSG_USERAUTH_REQUEST "id=5; username=MyUser; service=ssh-connection; method=password; password=*******" - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_USERAUTH_SUCCESS id=5 - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - "logged in" - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_MSG_CHANNEL_OPEN "id=6; type=session; sender_channel=1; initial_window_size=131072; mac_packet_size=32768" - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_OPEN_CONFIRMATION "id=6; recipient_channel=1; sender_channel=0; initial_window_size=32368; mac_packet_size=32368; data=" - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_MSG_CHANNEL_REQUEST "id=7; recipient_channel=0; type=subsystem; want_reply=true; subsystem=sftp" - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_SUCCESS "id=7; channel#=1" - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_MSG_CHANNEL_DATA "id=8; channel#=0; data_length=9; data=00:00:00:05:01:00:00:00:03..." - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_WINDOW_ADJUST "id=8; channel#=1; bytes=97113" - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_FXP_INIT " id=0; version=3" - - - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_DATA "id=9; channel#=1; data_length=9; data=00:00:00:05:02:00:00:00:03..." - -
2011-03-24 14:19:47 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_FXP_VERSION " id=0; version=3; extensions=()" - -
2011-03-24 14:19:49 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_MSG_CHANNEL_DATA "id=9; channel#=0; data_length=44; data=00:00:00:28:03:00:00:00:01:00..." - - - -
2011-03-24 14:19:49 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_FXP_OPEN "id=1; filename=/MyData/pf10068.dat; pflags=1100; atrrs=(size=null, uid=null, gid=null, permissions=436, atime=null, mtime=null)" - - - -
2011-03-24 14:19:49 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_DATA "id=10; channel#=1; data_length=41; data=00:00:00:25:65:00:00:00:01:00..." - -
2011-03-24 14:19:49 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_FXP_STATUS "id=1; code=4; message=error opening file; tag=en" - -
2011-03-24 14:19:54 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser SSH_MSG_CHANNEL_CLOSE "id=10; channel#=0" - - - -
2011-03-24 14:19:54 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_EOF "id=11; channel#=1" - -
2011-03-24 14:19:54 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - SSH_MSG_CHANNEL_CLOSE "id=12; channel#=1" - -
2011-03-24 14:19:54 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 MyUser - - "logged out" - - -
2011-03-24 14:19:54 xxx.xxx.xxx.80 1455 xxx.xxx.xxx.65 22 - - - "session closed; " - - -

with this code:

Code: Select all

bool __fastcall TSftpForm::UploadProgress(char *localfile)
{
  Scclrclasses::TBytes Handle;
TScSFTPFileOpenModes FOM;
  Scsftputils::TScSFTPAceFlags* AF =  new(Scsftputils::TScSFTPAceFlags);
  Scsftputils::TScSFTPFileAttributes* FA = new(Scsftputils::TScSFTPFileAttributes);
  AnsiString Directory;
  int i,handle,bytes,blocks,rest;
  long filelen;
#define BUFFER_SIZE 32768
  byte Puffer[BUFFER_SIZE];

  FtpError = false;
  handle = open(localfile, O_RDONLY|O_BINARY);
  if (handle 0)
  {
  ConnectToFtp();
  //the upload directory is one level up (to the download directory)
  if (MaschPar.lov[MaschPar.LovSelection].Directory.Pos("\") > 0)
    Directory = MaschPar.lov[MaschPar.LovSelection].Directory.SubString(1,MaschPar.lov[MaschPar.LovSelection].Directory.LastDelimiter("\"));
  else
    Directory = MaschPar.lov[MaschPar.LovSelection].Directory.SubString(1,MaschPar.lov[MaschPar.LovSelection].Directory.LastDelimiter("/"));
  try
  {
    try
    {
  FOM.Clear();
  if (FOM.Contains(foAppend))
    AddMessage("_foAppend_",(TColor) FTP_MEMO_COLOR_SUCCESS);
  if (FOM.Contains(foCreate))
    AddMessage("_foCreate_",(TColor) FTP_MEMO_COLOR_SUCCESS);
  FOM Permissions = Scsftputils::TScSFTPFilePermissions() ValidAttributes = FA->ValidAttributes OpenFile(Directory+MaschPar.lov[MaschPar.LovSelection].Output,
                                  FOM,FA);
  if (FtpError)
  {
    DisconnectFromFtp();
    return FtpError;
  }

    blocks = filelen / BUFFER_SIZE;
    rest = filelen % BUFFER_SIZE;
    for (i=0;iWriteFile(Handle,0,(void *)&Puffer[0],bytes);
    }
    if (rest!=0)
    {
      if ((bytes = read(handle, Puffer, rest)) != rest)
      {
        close(handle);
        DisconnectFromFtp();
        return true;
      }
      ScSFTPClient->WriteFile(Handle,0,(void *)&Puffer[0],bytes);
    }
    ScSFTPClient->SetAttributesByHandle(Handle,FA);
    ScSFTPClient->CloseHandle(Handle);
    }
    __except( EXCEPTION_EXECUTE_HANDLER )
    {
      FtpError = true;
    }
  }
  catch ( ... )
  {
    FtpError = true;
  }
  DisconnectFromFtp();
  }
  close(handle);
  return FtpError;
}

so, what's wrong?
Is this combination of OpenFile (foCreate, foAppend) with WriteFile ever tested/used by somebody? Is this a "standard" feature? Should this be supported by any SFTP server or is this maybe SFTP protocol version dependent?
Is there any (working) example showing the use of this function?
I NEED HELP ON THIS!
/gw

Dimon · Post by **Dimon** » Mon 28 Mar 2011 11:42

To solve the problem try opening a remote file with the Modes parameter equaled [foWrite, foAppend, foCreate].

gw · Post by gw » Tue 29 Mar 2011 08:39

Hi Dimon!

Thanks for your reply.
I changed code to this:

Code: Select all

  FOM.Clear(); 
  FOM << Scsftputils::foWrite << Scsftputils::foAppend << Scsftputils::foCreate;

Then i tested with these three SFTP-Servers:
1. "freeFTPd"
it still doesn't work
2. "Cerberus FTP Server"
there is no error but (usually) the file isn't appended but truncated (old content is gone). Yesterday it worked one time, but today i did some more tests and they all 'failed'.
3. "JSCAPE MFT Server"
*surprise* all tests worked well.

So i will check and see if customers (linux) server will work well also (but this may take some time, since the customer have put the conversion from FTP to SFTP temporarily on hold).

Thanks for your support so far.

best regards
/gw

Dimon · Post by **Dimon** » Wed 30 Mar 2011 09:57

We have investigated this problem. This behaviour depends on the SFTP server and is not connected with the SFTP client, therefore we can't influence it.
To solve the problem try using the following code:

Code: Select all

  Stream := TFileStream.Create(Source, fmOpenRead);
  try
    Attrs := TScSFTPFileAttributes.Create;
    Handle := ScSFTPClient.OpenFile(Destination, [foWrite, foCreate], nil);

    try
      ScSFTPClient.RetrieveAttributesByHandle(Attrs, Handle);
      if aSize in Attrs.ValidAttributes then
        FileOffset := Attrs.Size
      else
        FileOffset := 0;

      SetLength(Buf, Min(32768, Stream.Size));
      while Stream.Position  Stream.Size do begin
        Count := Stream.Read(Buf[0], Length(Buf));
        if Count = 0 then Exit;
        ScSFTPClient.WriteFile(Handle, FileOffset, Buf, 0, Count);
        FileOffset := FileOffset + Count;
      end;

    finally
      Attrs.Free;
      ScSFTPClient.CloseHandle(Handle);
    end;
  finally
    Stream.Free;
  end;

gw · Post by gw » Tue 05 Apr 2011 16:36

Nice piece of code

Here is my implementation in C (might be helpful for other C programmers in the future):

Code: Select all

  AnsiString Directory;
  Scclrclasses::TBytes Handle;
  TScSFTPFileOpenModes FOM;
  TBytes Buffer;
  __int64 FileOffset;
  int Buffersize;
  int Count;

  FtpError = false;
  //the upload directory is one level up (to the download directory)
  if (MaschPar.lov[MaschPar.LovSelection].Directory.Pos("\") > 0)
    Directory = MaschPar.lov[MaschPar.LovSelection].Directory.SubString(1,MaschPar.lov[MaschPar.LovSelection].Directory.LastDelimiter("\"));
  else
    Directory = MaschPar.lov[MaschPar.LovSelection].Directory.SubString(1,MaschPar.lov[MaschPar.LovSelection].Directory.LastDelimiter("/"));
  TFileStream *Stream = new TFileStream(AnsiString(localfile),fmOpenRead);
  try
  {
    if (Stream->Size > 0)
    {
    ConnectToFtp();
    if (FtpError)
    {
      delete Stream;
      DisconnectFromFtp();
      return FtpError;
    }
    Scsftputils::TScSFTPFileAttributes* FA = new(Scsftputils::TScSFTPFileAttributes);
    FOM.Clear();
    FOM OpenFile(Directory+MaschPar.lov[MaschPar.LovSelection].Output,
                                    FOM,NULL);
    try
    {
      ScSFTPClient->RetrieveAttributesByHandle(FA, Handle, TScSFTPAttributes() ValidAttributes.Contains(aSize))
        FileOffset = FA->Size;
      else
        FileOffset = 0;
      Buffersize = min(ScSFTPClient->ReadBlockSize,Stream->Size);
      Buffer.Length = Buffersize;
      try
      {
        while (Stream->Position != Stream->Size)
        {
          Count = Stream->Read(&Buffer[0], Buffersize);
          if (Count == 0)
            break;
          ScSFTPClient->WriteFile(Handle,FileOffset,Buffer,0,Count);
          FileOffset = FileOffset + Count;
        }
      }
      __finally
      {
        Buffer.Length = 0;
      }
    }
    __finally
    {
      delete FA;
      ScSFTPClient->CloseHandle(Handle);
      DisconnectFromFtp();
    }
    }
  }
  __finally
  {
    delete Stream;
  }
  return FtpError;

Successfully tested with these four SFTP-Servers:
1. "freeFTPd"
2. "Cerberus FTP Server"
3. "JSCAPE MFT Server"
4. "Core FTP mini SFTP Server"

Test with customers linux SFTP-server will be sometime in the future.

Thanks for your support

/gw

Dimon · Post by **Dimon** » Wed 06 Apr 2011 06:57

It is good to see that this problem has been solved. If any other questions come up, please contact me.