Unable to process utf8mb4 data

Discussion of open issues, suggestions and bugs regarding usage of dbExpress drivers for MySQL in Delphi and C++Builder
Post Reply
kkkelvinkk
Posts: 2
Joined: Mon 11 Jan 2016 03:44

Unable to process utf8mb4 data

Post by kkkelvinkk » Mon 11 Jan 2016 07:15

I am trying to use dbexpress driver for mysql to read and write utf8mb4 data to mysql. I tried both XE2 and XE8, but the driver is not able to update utf8mb4 data into mysql using the option 'use unicode=true'. Moreover, there is no parameter to set charset for the driver. For example, the chinese character (http://www.scarfboy.com/coding/unicode-tool?s=U%2B232ab) is only available in utf8mb4 but not in utf8.

Is there any workaround or solutions ?

I cannot post the character to the forum. So, please refer to the link for that chinese character.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Unable to process utf8mb4 data

Post by ViktorV » Tue 12 Jan 2016 14:50

Unfortunately, we couldn't reproduce the issue: data in utf8mb4 encoding was inserted and updated correctly.
Please check whether the problem is reproduced on our Query demo project. You can find the demo project in the %DBXMySQLDemos%\Win32\Query directory, where %DBXMySQL% is the path to the installed dbExpress driver for MySQL Demo projects installation path on your computer.
If the problem is not reproduced on our demo project, please send us a small sample to demonstrate the issue, including scripts for creating database objects.

kkkelvinkk
Posts: 2
Joined: Mon 11 Jan 2016 03:44

Re: Unable to process utf8mb4 data

Post by kkkelvinkk » Wed 13 Jan 2016 06:23

Yes. The problem is exists in the sample project.
Below is the script to create the database.
CREATE SCHEMA `test1` DEFAULT CHARACTER SET utf8 ;

CREATE TABLE `test1`.`dept` (
`name` VARCHAR(100) CHARACTER SET 'utf8mb4' NULL);

ALTER TABLE `test1`.`dept`
CHARACTER SET = utf8 ;

grant all privileges on test1.* to testuser@localhost identified by 'testpassword';
flush privileges;

Please test the character in this link: http://www.scarfboy.com/coding/unicode-tool?s=U%2B232ab
As I cannot post this character in your forum, I think this character also cannot insert into your forum database.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Unable to process utf8mb4 data

Post by ViktorV » Fri 15 Jan 2016 11:09

Thank you for the information. The problem investigation is in progress. We will inform you when we have any results.

kdjfakej
Posts: 2
Joined: Fri 04 Sep 2015 04:46

Re: Unable to process utf8mb4 data

Post by kdjfakej » Sat 16 Apr 2016 07:11

This is a major issue. Current MySQL driver implementation DOES NOT account for this charset (utf8mb4), it is simply not supported. First, there is a constant limit of MaxChars in UTF set to 3. Second, the connection issues command 'SET NAMES utf8' that makes it impossible to use utf8mb4 -- there is an error when the 4-byte UTF symbols are inserted into the utf8mb4 database/table/column. I was able to manually enter utf8mb4 into the SET NAMES command in the source codes and fix some comparisons to include utf8mb4 charset in some of the MySQL units, which stopped this error. However, still, the high surrogates become corrupt (due to the fact that the internal functions do not support them when converting this from Delphi's UTF-16, generating garbage). Please fix ASAP - this IS a bug as it utf8mb4 is the only format that fully supports the entire UTF universe, and the simple UTF8 usage is NOT recommended for MySQL deployments.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Unable to process utf8mb4 data

Post by ViktorV » Wed 27 Apr 2016 08:57

We have added support for utf8mb4 charset. It will be included in the next build of dbExpress Driver for MySQL that we are going to release within a week.

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Unable to process utf8mb4 data

Post by ViktorV » Thu 28 Apr 2016 14:21

The new dbExpress Driver for MySQL 6.8.11 with support for utf8mb4 charset is already available for download now.

Post Reply