About the surrogate pair correspondence of Unicode

Discussion of open issues, suggestions and bugs regarding IBDAC (InterBase Data Access Components) for Delphi, C++Builder, Lazarus (and FPC)
Post Reply
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

About the surrogate pair correspondence of Unicode

Post by tom1122 » Thu 08 Jan 2009 10:26

hi.

The error occurred by the surrogate pair character.

The test environment is
C++Builder2009Pro
IBDAC 2.70.0.44
Firebird2.1(CHARSET UNICODE_FSS)

The tested character is　JIS X 0213/2004 U+20B9F.

By the way, it is possible to display it in IBX :shock:
Please help.

The error message is
Utf8ToWs:Malformed trail byte (D:\projects\Delphi\Dac\Source\MemUtils.pass,1246)

[12965CD2]{dac120.bpl } Memutils.Utf8ToWs + $10A
[128D561D]{ibdac120.bpl} Ibcclasses.TGDSRecordSet.GetComplexFldOffset + $811
[1296A73E]{dac120.bpl } Memdata.TFieldDescs.GetItems + $A
[1296E035]{dac120.bpl } Memdata.TData.HasFields + $49
[128D5ECC]{ibdac120.bpl} Ibcclasses.TGDSRecordSet.InternalFetch + $5C8
[12A32349]{dacvcl120.bpl} Dacvcl.StartWait + $5
[128D6441]{ibdac120.bpl} Ibcclasses.TGDSRecordSet.Fetch + $65
[1297E37E]{dac120.bpl } Craccess.TCRRecordSet.GetNextRecord + $26
[129891DE]{dac120.bpl } Memds.TMemDataSet.GetRecord + $62
[50496298]{dbrtl120.bpl} DB.TDataSet.GetNextRecord (Line 11640, "DB.pas" + 9) + $14
[50496619]{dbrtl120.bpl} DB.TDataSet.GetNextRecords (Line 11775, "DB.pas" + 2) + $D
[504960D2]{dbrtl120.bpl} DB.TDataSet.SetBufferCount (Line 11569, "DB.pas" + 25) + $5
[5049616A]{dbrtl120.bpl} DB.TDataSet.UpdateBufferCount (Line 11594, "DB.pas" + 15) + $4
[504941C8]{dbrtl120.bpl} DB.TDataSet.DoInternalOpen (Line 10601, "DB.pas" + 4) + $2
[50494272]{dbrtl120.bpl} DB.TDataSet.OpenCursor (Line 10628, "DB.pas" + 4) + $2
[12987336]{dac120.bpl } Memds.TMemDataSet.OpenCursor + $A
[1299E082]{dac120.bpl } Dbaccess.TCustomDADataSet.OpenCursor + $1A6
[5049412D]{dbrtl120.bpl} DB.TDataSet.SetActive (Line 10581, "DB.pas" + 12) + $7
[1299DEC9]{dac120.bpl } Dbaccess.TCustomDADataSet.SetActive + $3D
[5003F57A]{rtl120.bpl } TypInfo.SetOrdProp (Line 1502, "TypInfo.pas" + 27) + $0
[20B9ACF2]{designide120.bpl} DesignEditors.TPropertyEditor.SetOrdValue (Line 838, "DesignEditors.pas" + 2) + $E
[20B9BDFB]{designide120.bpl} DesignEditors.TEnumProperty.SetValue (Line 1398, "DesignEditors.pas" + 5) + $4
[20B9A961]{designide120.bpl} DesignEditors.TPropertyEditor.Edit (Line 668, "DesignEditors.pas" + 11) + $14
[208AAFDA]{coreide120.bpl} PropInsp.TPropertyInspector.PropListEditDblClick (Line 882, "PropInsp.pas" + 20) + $5
[20C87371]{vclide120.bpl} IDEInspListBox.TInspListBox.DoEditDblClick (Line 976, "IDEInspListBox.pas" + 2) + $A
[20C88BB2]{vclide120.bpl} IDEInspListBox.TInspListBox.EditDblClick (Line 1546, "IDEInspListBox.pas" + 0) + $2
[501D1195]{vcl120.bpl } Controls.TControl.DblClick (Line 6763, "Controls.pas" + 1) + $14
[501D12F8]{vcl120.bpl } Controls.TControl.WMLButtonDblClk (Line 6806, "Controls.pas" + 4) + $C
[501D0C0E]{vcl120.bpl } Controls.TControl.WndProc (Line 6642, "Controls.pas" + 91) + $6
[501D529B]{vcl120.bpl } Controls.TWinControl.WndProc (Line 9336, "Controls.pas" + 136) + $6
[501DD25A]{vcl120.bpl } Controls.TDockTree.WndProc (Line 14469, "Controls.pas" + 74) + $5
[500477CC]{rtl120.bpl } Classes.TThreadList.UnlockList (Line 3781, "Classes.pas" + 0) + $4
[5012C8FC]{vcl120.bpl } Graphics.FreeMemoryContexts (Line 6830, "Graphics.pas" + 12) + $5
[501D49B4]{vcl120.bpl } Controls.TWinControl.MainWndProc (Line 9065, "Controls.pas" + 3) + $6
[501D4BF4]{vcl120.bpl } Controls.TWinControl.IsControlMouseMsg (Line 9129, "Controls.pas" + 9) + $25
[501D529B]{vcl120.bpl } Controls.TWinControl.WndProc (Line 9336, "Controls.pas" + 136) + $6
[20C86513]{vclide120.bpl} IDEInspListBox.TPropInspEdit.WndProc (Line 439, "IDEInspListBox.pas" + 7) + $4
[501D49B4]{vcl120.bpl } Controls.TWinControl.MainWndProc (Line 9065, "Controls.pas" + 3) + $6
[50058938]{rtl120.bpl } Classes.StdWndProc (Line 12723, "Classes.pas" + 8) + $0
[501FB05D]{vcl120.bpl } Forms.TApplication.StopHintTimer (Line 10343, "Forms.pas" + 3) + $6
[501F9EC7]{vcl120.bpl } Forms.TApplication.ProcessMessage (Line 9613, "Forms.pas" + 23) + $1
[501F9F0A]{vcl120.bpl } Forms.TApplication.HandleMessage (Line 9643, "Forms.pas" + 1) + $4
[501FA235]{vcl120.bpl } Forms.TApplication.Run (Line 9780, "Forms.pas" + 26) + $3
[004259D2]{bds.exe } bds.bds (Line 196, "" + 7) + $7
Top
Plash
Devart Team
Posts: 2844
Joined: Wed 10 May 2006 07:09

Post by Plash » Fri 09 Jan 2009 10:33

Please describe how to insert this character into the database.
Top
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

Post by tom1122 » Fri 09 Jan 2009 17:57

The following SQL was executed by using FlameRobin0.9

Code: Select all

CREATE TABLE TEST (
    CD    INTEGER NOT NULL,
    NAME  CHAR(10) NOT NULL
);

INSERT INTO TEST VALUES(1, '𠮟' )
Insert was possible also by TIBCSQL.

Code: Select all

IBCSQL1->SQL->Add(L"insert into test values(1, '𠮟' )");
IBCSQL1->Execute();
It is URL of the surrogate pair list.
http://w3.kcua.ac.jp/~fujiwara/jis2000/ ... surro.html

Can you do cut and paste from this homepage?
Cannot you display for a Japanese font not to exist?
Top
Plash
Devart Team
Posts: 2844
Joined: Wed 10 May 2006 07:09

Post by Plash » Tue 13 Jan 2009 08:55

We cannot see the characters on this page. Please give us encoded value of the surrogate pair by using the following code (replace 'a' with your character):

Code: Select all

var
  UTF, Buf: AnsiString;
begin
  UTF := UTF8Encode('a');
  SetLength(Buf, Length(UTF) * 2);
  BinToHex(PAnsiChar(UTF), PAnsiChar(Buf), Length(UTF));
  ShowMessage(Buf);
end;
Top
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

Post by tom1122 » Wed 14 Jan 2009 11:10

Because it is not possible to compile in AnsiString, UnicodeString is used.
Is it good in this?

Code: Select all

	UnicodeString a;
	UnicodeString UTF;
	UnicodeString Buf;

	a = L"𠮟";
	UTF = UTF8Encode(a);
	Buf.SetLength(UTF.Length() * 2);
	BinToHex(UTF.c_str(), Buf.c_str(), UTF.Length() );
	ShowMessage(Buf);
The displayed message is 42D8.
The FDB file is sent if necessary.
Top
Plash
Devart Team
Posts: 2844
Joined: Wed 10 May 2006 07:09

Post by Plash » Thu 15 Jan 2009 08:52

This code is not correct. You can use the following code:

Code: Select all

  AnsiString Buf;
  RawByteString UTF;
  UTF = UTF8Encode(WideString("a"));
  Buf.SetLength(UTF.Length() * 2);
  BinToHex(UTF.c_str(), Buf.c_str(), UTF.Length());
  ShowMessage(Buf);
Or just send to ibdac*devart*com the FDB file.
Top
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

Post by tom1122 » Fri 16 Jan 2009 01:49

The displayed message is F0A0AE9F.

The FDB file is sent.
Please confirm it.
Top
Plash
Devart Team
Posts: 2844
Joined: Wed 10 May 2006 07:09

Post by Plash » Fri 16 Jan 2009 12:09

We have reproduced this problem, and we are working on it. You will be notified when we fix the problem.
Top
Plash
Devart Team
Posts: 2844
Joined: Wed 10 May 2006 07:09

Post by Plash » Wed 21 Jan 2009 12:32

We have fixed this problem. The fix will be included in the next build of IBDAC.
Top
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

Post by tom1122 » Thu 22 Jan 2009 00:28

Thank you.
I am looking forward to the next build of IBDAC.
Top
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

Post by tom1122 » Tue 15 Sep 2009 10:45

Hi.

Though time passed considerably...
There is no problem in CHARSET UNICODE_FSS.

The same table was made by CHARSET UTF8.

It garbles when displaying it with DBGRID.

In UNICODE_FSS and UTF8, is processing different?

The test environment is
C++Builder2009Pro
IBDAC 3.10.0.7
Firebird2.1.3(CHARSET UTF8)
Top
Plash
Devart Team
Posts: 2844
Joined: Wed 10 May 2006 07:09

Post by Plash » Wed 16 Sep 2009 11:11

Processing is not different. We cannot reproduce the problem. Please send to support*devart*com a complete small sample that demonstrates the problem, including the script for creating database objects.
Top
tom1122
Posts: 8
Joined: Thu 08 Jan 2009 10:02

Post by tom1122 » Fri 18 Sep 2009 08:16

Fdb and the source were sent with mail.
Top
Post Reply

Return to “InterBase Data Access Components”