UniDAC/MySQL Provider - Error with MySQL INSERT statement relating to unicode characters

Discussion of open issues, suggestions and bugs regarding UniDAC (Universal Data Access Components) for Delphi, C++Builder, Lazarus (and FPC)
Post Reply
stevel
Posts: 125
Joined: Tue 02 Nov 2010 19:01

UniDAC/MySQL Provider - Error with MySQL INSERT statement relating to unicode characters

Post by stevel » Wed 17 Feb 2021 10:40

Hello,

Using: UniDAC 8.1.2, Delphi 10.2.3 Tokyo, MySQL Community Server 8.0.2 running on Windows 10 x64

My data transfer program is transferring data from SQL Server to MySQL, using 2 TUniConnection and TUniQuery instances - one to select from source database (MS-SQL 2014) and the other to insert into destination database (MySQL 8). When inserting rows into the MySQL database, I'm getting the following error:
---------------------------
Debugger Exception Notification
---------------------------
Project xxxxxxx.exe raised exception class EMySqlException with message '#HY000Incorrect string value: '\xE9 uits...' for column 'description' at row 1'.
---------------------------
Break Continue Help
---------------------------

The MySQL database and table description is below:

Code: Select all

CREATE DATABASE `testdb` /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci */ 
/*!80016 DEFAULT ENCRYPTION='N' */;

CREATE TABLE `emails_text` (
  `email_id` char(36) NOT NULL,
  `description` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `deleted` tinyint(1) DEFAULT '0',
  PRIMARY KEY (`email_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;


The code I'm using to initialize the connection, and the queries is as below:

Code: Select all

var
  mUniconSrcDB, mUniconDstDB: TUniConnection;
  uqSrc, uqDst: TUniQuery;
begin
  mUniconSrcDB := TUniConnection.Create(nil);
  mUniconSrcDB.ProviderName := 'SQL Server';
  mUniconSrcDB.Server := 'xxxxxx';
  mUniconSrcDB.Database := 'xxxxxx';
  mUniconSrcDB.Username := 'xxxxxx';
  mUniconSrcDB.Password := 'xxxxxx';
  try
    mUniconSrcDB.Connect;
  except
  end;

  mUniconDstDB := TUniConnection.Create(nil);
  mUniconDstDB.ProviderName := 'MySQL';
  mUniconDstDB.SpecificOptions.Values['UseUnicode'] := 'True';
  mUniconDstDB.Server := 'xxxxxx';
  mUniconDstDB.Database := 'xxxxxx';
  mUniconDstDB.Username := 'xxxxxx';
  mUniconDstDB.Password := 'xxxxxx';
  try
    mUniconDstDB.Connect;
  except
  end;


  uqDst := TUniQuery.Create(nil);
  uqDst.Connection := mUniconDstDB;

  uqDst.SQL.Text := 'INSERT INTO emails_text (email_id, description, deleted ) ' +
        ' VALUES (:EMAIL_ID, :DESCRIPTION, :DELETED ) ';
  uqDst.Prepare;
  uqDst.ParamByName('EMAIL_ID').AsString := uuid_email; //
  uqDst.ParamByName('DESCRIPTION').AsMemo := TRTF.Decode(uqSrc.FieldByName('DETAILS').AsString); //
  uqDst.ParamByName('DELETED').AsBoolean := False; //
  try
    uqDst.ExecSQL;

  except
    on ex: Exception do
    begin
      Inc(e);
      if (mLogFile <> EmptyStr) then
        LogToFile(mLogFile, 'uqDst ' + ex.Message);
    end;
  end;
  uqDst.Close;
 

class function TRTF.CreateRichEdit: TRichEdit;
begin
  Result := TRichEdit.CreateParented(HWND_MESSAGE);
end;

class function TRTF.Encode(const AInput: string): string;
var
  RichEdit: TRichEdit;
  Stream: TStringStream;
begin
  RichEdit := CreateRichEdit;
  try
    RichEdit.Lines.Text := AInput;
    Stream := TStringStream.Create;
    try
      RichEdit.Lines.SaveToStream(Stream);
      Result := Stream.DataString;
    finally
      Stream.Free;
    end;
  finally
    RichEdit.Free;
  end;
end;

class function TRTF.Decode(const AInput: string): string;
var
  RichEdit: TRichEdit;
  Stream: TStringStream;
begin
  RichEdit := CreateRichEdit;
  try
    Stream := TStringStream.Create(AInput);
    try
      RichEdit.Lines.LoadFromStream(Stream);
      Result := RichEdit.Lines.Text;
    finally
      Stream.Free;
    end;
  finally
    RichEdit.Free;
  end;
end;
I would really appreciate if you could help me resolve this error or point me in the right direction! TIA

stevel
Posts: 125
Joined: Tue 02 Nov 2010 19:01

Re: UniDAC/MySQL Provider - Error with MySQL INSERT statement relating to unicode characters

Post by stevel » Wed 17 Feb 2021 11:30

Thanks to StackOverflow user @electrobabe, I found how to resolve this issue via setting MySQL server session parameters via code.

The change needed to resolve the issue is:

Change

Code: Select all

  try
    mUniconDstDB.Connect;
  except
  end;
to

Code: Select all

  try
    mUniconDstDB.Connect;

    mUniconDstDB.ExecSQL('set character_set_server = utf8mb4;');
    mUniconDstDB.ExecSQL('set collation_server = utf8mb4_general_ci;');
    mUniconDstDB.ExecSQL('set character_set_connection = utf8mb4;');
    mUniconDstDB.ExecSQL('set collation_connection = utf8mb4_general_ci;');
    mUniconDstDB.ExecSQL('set character_set_database = utf8mb4;');
    mUniconDstDB.ExecSQL('set collation_database = utf8mb4_general_ci;');
  except
  end;

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: UniDAC/MySQL Provider - Error with MySQL INSERT statement relating to unicode characters

Post by ViktorV » Fri 19 Feb 2021 12:41

Unfortunately, we cannot reproduce the issue using the information provided by you on the latest version UniDAC 8.3.2.
Please check if the behavior you specified reproduces when using the latest version of UniDAC 8.3.2 available for download on our website. If reproduced on the latest version, please specify the value of the DESCRIPTION parameter, when inserted, the error you specified will be raised.

Post Reply