Page 1 of 1

Unable to process utf8mb4 data

Posted: Mon 11 Jan 2016 07:15
by kkkelvinkk
I am trying to use dbexpress driver for mysql to read and write utf8mb4 data to mysql. I tried both XE2 and XE8, but the driver is not able to update utf8mb4 data into mysql using the option 'use unicode=true'. Moreover, there is no parameter to set charset for the driver. For example, the chinese character (http://www.scarfboy.com/coding/unicode-tool?s=U%2B232ab) is only available in utf8mb4 but not in utf8.

Is there any workaround or solutions ?

I cannot post the character to the forum. So, please refer to the link for that chinese character.

Re: Unable to process utf8mb4 data

Posted: Tue 12 Jan 2016 14:50
by ViktorV
Unfortunately, we couldn't reproduce the issue: data in utf8mb4 encoding was inserted and updated correctly.
Please check whether the problem is reproduced on our Query demo project. You can find the demo project in the %DBXMySQLDemos%\Win32\Query directory, where %DBXMySQL% is the path to the installed dbExpress driver for MySQL Demo projects installation path on your computer.
If the problem is not reproduced on our demo project, please send us a small sample to demonstrate the issue, including scripts for creating database objects.

Re: Unable to process utf8mb4 data

Posted: Wed 13 Jan 2016 06:23
by kkkelvinkk
Yes. The problem is exists in the sample project.
Below is the script to create the database.
CREATE SCHEMA `test1` DEFAULT CHARACTER SET utf8 ;

CREATE TABLE `test1`.`dept` (
`name` VARCHAR(100) CHARACTER SET 'utf8mb4' NULL);

ALTER TABLE `test1`.`dept`
CHARACTER SET = utf8 ;

grant all privileges on test1.* to testuser@localhost identified by 'testpassword';
flush privileges;

Please test the character in this link: http://www.scarfboy.com/coding/unicode-tool?s=U%2B232ab
As I cannot post this character in your forum, I think this character also cannot insert into your forum database.

Re: Unable to process utf8mb4 data

Posted: Fri 15 Jan 2016 11:09
by ViktorV
Thank you for the information. The problem investigation is in progress. We will inform you when we have any results.

Re: Unable to process utf8mb4 data

Posted: Sat 16 Apr 2016 07:11
by kdjfakej
This is a major issue. Current MySQL driver implementation DOES NOT account for this charset (utf8mb4), it is simply not supported. First, there is a constant limit of MaxChars in UTF set to 3. Second, the connection issues command 'SET NAMES utf8' that makes it impossible to use utf8mb4 -- there is an error when the 4-byte UTF symbols are inserted into the utf8mb4 database/table/column. I was able to manually enter utf8mb4 into the SET NAMES command in the source codes and fix some comparisons to include utf8mb4 charset in some of the MySQL units, which stopped this error. However, still, the high surrogates become corrupt (due to the fact that the internal functions do not support them when converting this from Delphi's UTF-16, generating garbage). Please fix ASAP - this IS a bug as it utf8mb4 is the only format that fully supports the entire UTF universe, and the simple UTF8 usage is NOT recommended for MySQL deployments.

Re: Unable to process utf8mb4 data

Posted: Wed 27 Apr 2016 08:57
by ViktorV
We have added support for utf8mb4 charset. It will be included in the next build of dbExpress Driver for MySQL that we are going to release within a week.

Re: Unable to process utf8mb4 data

Posted: Thu 28 Apr 2016 14:21
by ViktorV
The new dbExpress Driver for MySQL 6.8.11 with support for utf8mb4 charset is already available for download now.