Charset NONE to UTF8, TCRBatchMove => "malformed string"

Discussion of open issues, suggestions and bugs regarding IBDAC (InterBase Data Access Components) for Delphi, C++Builder, Lazarus (and FPC)
Post Reply
Fabrice
Posts: 45
Joined: Tue 07 Sep 2010 09:44

Charset NONE to UTF8, TCRBatchMove => "malformed string"

Post by Fabrice » Fri 09 Sep 2016 10:05

Hello,


I'am trying to convert a database with charset None to the same but in UTF8.
Source database = Firebird 3.0 charset None Dialect 1
Destination database = Firebirde 3.0 charset UTF8 Dialect 1

I use a TCRBatchMove component to do that.
But each time a field of the source database containing a special char like "é" or "à" I receive an "Malformed string" error and importation is aborted.

That I have tried :
1) Source database force to with TIBCConnection.Options.Charset=Win1252
2) Destination database TIBCConnection..Options.useUnicode=true or false
3) Destination database TIBCConnection.Options.Charset=NONE
4) This viewtopic.php?f=24&t=24630&p=83281
5) Trying to do :

Code: Select all

procedure TDataCharsetNoneToUTF8.Q_DestinationAfterOpen(DataSet: TDataSet);
var
  i: Integer;
begin

  for i := 0 to DataSet.FieldCount - 1 do
    if DataSet.Fields[i] is TStringField then begin
      DataSet.Fields[i].OnGetText := OnGetText;
      DataSet.Fields[i].OnSetText := OnSetText;
    end;
end;
Even with combined both these change it's didn't worked.

My code is very simple and do something like that :

Code: Select all

       
    CRBatchMove: TCRBatchMove;
    Q_Source: TIBCQuery;
    Q_Destination: TIBCQuery;
    Base_Source: TIBCConnection;
    Base_DestinationUTF8: TIBCConnection;

.../...
function TDataCharsetNoneToUTF8.GetTableList: Tstrings;
begin
 if not assigned(FTableList) then
     FTableList:=TStringList.Create;

   if FTableList.count=0 then
   begin
     Base_Source.GetTableNames(FTableList);
   end;
  result:= FTableList;
end;

.../...
begin
for I := 0 to FTableList.Count-1 do
       begin
         Q_Source.Active:=false;
         Q_destination.Active:=false;

         Q_Source.MacroByName('TableName').Value:=FTableList[i];

         Q_SourceCompte.Active:=false;
         Q_SourceCompte.MacroByName('TableName').Value:=FTableList[i];
         Q_SourceCompte.Active:=true;

         Q_destination.MacroByName('TableName').Value:=FTableList[i];


              try
                  Q_Source.Active:=True;
                  Q_destination.Active:=True;
                  CRBatchMove.Execute;
                  Q_destination.Transaction.CommitRetaining;



              except

              end;






       end;
end;
with

Code: Select all

object CRBatchMove: TCRBatchMove
    AbortOnKeyViol = False
    AbortOnProblem = False
    Destination = Q_Destination
    Source = Q_Source
    OnBatchMoveProgress = CRBatchMoveBatchMoveProgress
    Left = 208
    Top = 176
  end
  object Q_Source: TIBCQuery
    Connection = Base_Source
    SQL.Strings = (
      'select * from &TableName')
    AfterOpen = Q_DestinationAfterOpen
    Left = 128
    Top = 176
    MacroData = <
      item
        Name = 'TableName'
      end>
  end
  object Q_Destination: TIBCQuery
    Connection = Base_DestinationUTF8
    SQL.Strings = (
      'select * from &TableName '
      'where (0=1)')
    Options.StrictUpdate = False
    BeforePost = Q_DestinationBeforePost
    OnPostError = Q_DestinationPostError
    Left = 288
    Top = 176
    MacroData = <
      item
        Name = 'TableName'
      end>
  end
  object Base_Source: TIBCConnection
    Database = 'H:\data\DATABASESOURCE_NONE.FDB'
    Options.Charset = 'NONE'
    SQLDialect = 1
    ClientLibrary = 'gds32.dll'
    Port = '3050'
    Username = 'sysdba'
    Server = '192.168.1.101'
    LoginPrompt = False
    Left = 128
    Top = 96
  end
  object Base_DestinationUTF8: TIBCConnection
    Database = 'H:\data\DATABASESOURCE_UTF8.FDB'
    Options.Charset = 'UTF8'
    Options.UseUnicode = True
    Options.NoDBTriggers = True
    SQLDialect = 1
    ClientLibrary = 'gds32.dll'
    Port = '3050'
    Username = 'sysdba'
    Server = '192.168.1.101'
    LoginPrompt = False
    Left = 288
    Top = 96
  end

Maybe a problem with Firebird 3 ?
I use version 5.7.24 on Delphi XE.

Best regards,
Fabrice

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Charset NONE to UTF8, TCRBatchMove => "malformed string"

Post by ViktorV » Fri 09 Sep 2016 12:11

Please compose a small sample demonstrating the described behavior and send it to viktorv*devart*com, including scripts creating database objects in order to get a detailed answer.

Fabrice
Posts: 45
Joined: Tue 07 Sep 2010 09:44

Re: Charset NONE to UTF8, TCRBatchMove => "malformed string"

Post by Fabrice » Mon 12 Sep 2016 13:24

Hello,

Maybe this will help :
The problem occurs with "BLOB SUB_TYPE 1" field which contains accent car like "é","è" "à" or "€". but not ion Varchar fields.
I will send you a full sample.

I haven't tested on Firebird 2.5...


Best regards,
Fabrice

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Charset NONE to UTF8, TCRBatchMove => "malformed string"

Post by ViktorV » Wed 14 Sep 2016 08:12

To solve the problem, try to set the TIBCConnection.Options.EnableMemos and TIBCConnection.Options.UseUnicode properties to True for destination database.

Fabrice
Posts: 45
Joined: Tue 07 Sep 2010 09:44

Re: Charset NONE to UTF8, TCRBatchMove => "malformed string"

Post by Fabrice » Mon 19 Sep 2016 09:02

Hello,
ViktorV wrote:To solve the problem, try to set the TIBCConnection.Options.EnableMemos and TIBCConnection.Options.UseUnicode properties to True for destination database.
Set TIBCConnection.Options.EnableMemos in both source and destination has fixed the problem.


Thank you.
Best regards,

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Charset NONE to UTF8, TCRBatchMove => "malformed string"

Post by ViktorV » Mon 19 Sep 2016 12:08

It is good to see that the issue has been solved.
Feel free to contact us if you have any further questions about our products.

Post Reply