Devart Forums

Posted: **Fri 10 Dec 2010 04:56**

Thanks for the new version for fixing unicode issue. However, it introduce new issues regarding blob field

CREATE TABLE `testunicode` (
`ID` int(11) NOT NULL,
`latin1` text,
`big5` text,
`gbk` text,
`png` blob,
PRIMARY KEY (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

The following is generated by dbForge, which is correct :
INSERT INTO testunicode VALUES
(0, 'E B S S y s t e m', '永富商業系統', '永富商业系统',

The Following is generated by TMyDump (Hex Blob = False) :

INSERT INTO testunicode VALUES
(0, 'E B S S y s t e m', '永富商業系統', '永富商业系统', 'NG\r\n\Z\n\0\0\0\rIHDR\0\0\0\0\0\0\0\0\0?a\0\0\0tE?vN\f0鵨*??`yiI?路?C殌V髶:?m?M\砥e$&^ju窙荔走;c媩?黏?? Z挌?齀繠?撐 洪?猷?gB{\Y?籮~#?揇帤T洎?M:尚*?-goc鍭偷N?A裫{P]S3.?0TW槲塤[4?}?瀔}?R都TVJh阿~蕞杓O?6才??橖芋陣?Q~睛璞n.蕨柆?j!r獦\[??l粖?摑{?Z?B涳c敗+留&?壯皚?W詄潁uLE岌#Js?\0$c\"壴g\0\0\0\0IEND唇`?);

(remarks : no ending quote)

The Following is generated by TMyDump (Hex Blob = True) :

INSERT INTO testunicode VALUES
(0, 'E B S S y s t e m', '永富商業䑄㕅㥆㔷䌴㔴㝁㑃㌲䄴㌷㐸䘳䙆㔰㠱〰㐲䙆㌶㈲䕃䘵㜶ㄱ〰〰〰〰㤴㔴䔴㐴䕁㈴〶㈸

Another minor issue is that it use unicode (Codepage 1200) which other use utf-8 (CodePage 65001). So it generate file 2X size of utf-8. It is not a very big deal though.

I don't know if there is other issues for now.

I can't wait another month for you next release so can I email you my customer info and ask for giving me the updated source (relevant file only) for my own re-compilation and further testing?

Posted: **Fri 10 Dec 2010 07:13**

I think I had find a fix in HexBlob mode.

In MyService.TCustomMyDumpProcessor.BackupObjects's
Sub Procedure ProcessField Line 1641
You still using PAnsiChar Typecast which should be PWideChar for VER12P

So, I changed the PAnsiChar to {$IFDEF VER12P}PWideChar{$ELSE}PAnsiChar{$ENDIF}

and the generated result is OK.

I don't know how to fix the problem is HexBlob mode is false.

Also, I had many blob fields which actually has strings and I see that that procedure also change those strings to Hex is make the file even larger. It is not critical though as at least it work.

Thanks again for your effort of fixing issues.

Posted: **Fri 10 Dec 2010 10:04**

Regarding using utf8 to reduce file size, I can make it work successfully. The resulting file size is slightly over 50% of using Unicode.

For my modification, Backup part is easy and straight forward :

Code: Select all

MyService.TCustomMyDumpProcessor.Backup
Change 
s := #$FF#$FE;
to
s := #$EF#$BB#$BF;

MyService.TCustomMyDumpProcessor.Add
Change
buf := Encoding.Unicode.GetBytes(WideString(Line + #$D#$A));
to
buf := Encoding.UTF8.GetBytes(UTF8Encode(Line + #$D#$A));

For Restore, I make the following change initially and it work in restoring the backup.

Code: Select all

DAScript.TDAScriptProcessor.CreateParser
Line 400 Change
enc := Encoding.Default;
to
enc := Encoding.UTF8;

However, it cause a index out of range error after the successful restore. With a closer look, the TParser take the stream size as TextLength but those multiple bytes char length 1 in FBlockSize but actually take more then 1 bytes. So, at the end, TextLength is bigger then FBlockSize and the Parser ReadNextBlock (reading nothing) and finally read from the empty buffer and cause index out of range.

So I make the following change :

Code: Select all

CRParser.TParser.ReadNextBlock;
Line 1076 Add
if Size  FBlockSize then // i.e. Variable Length char included
  TextLength := TextLength - (Size - FBlockSize);

I actually don't know if this code is appropriate as I am not good at stream management and parsing. However, it work well enough in my own case and prevent the out of range error. I hope your team can do better modification for this issue as saving almost 50% file size is a big plus.

Posted: **Fri 10 Dec 2010 10:28**

as the display is ruined by those long lines, please read the updated post instead :

http://www.devart.com/forums/viewtopic.php?t=19739

Posted: **Fri 10 Dec 2010 12:23**

Thank you for information. We have reproduced this problem and fixed it. This fix will be included in the next MyDAC build.

Devart Forums

New Problems of TMyDump of 6.00.0.3

New Problems of TMyDump of 6.00.0.3