Error when inserting large binary files using EF4

[asj]

Hello Devart,

I am experiencing problems when inserting binary files into a postgres bytea column using Entity Framework 4 and dotConnect. When inserting a file i get either a System.InvalidOperationException("Connection must be opened.") (150 MB file) or a System.OutOfMemoryException (200+ MB file).

I have a test project available demonstrating the behavior. Do you have a place where i may upload it?


Full tracebacks for the Exceptions are as follows:
System.InvalidOperationException
Message: "Connection must be opened."

StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25


System.OutOfMemoryException:
Message: "Exception of type 'System.OutOfMemoryException' was thrown."

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Devart.Data.PostgreSql.o.a(Byte[] A_0, Int32 A_1, Int32 A_2, Int32 A_3)
at Devart.Data.PostgreSql.o.b(Byte[] A_0, Int32 A_1, Int32 A_2)
at Devart.Data.PostgreSql.o.a(Char A_0)
at Devart.Data.PostgreSql.s.b(Char A_0)
at Devart.Data.PostgreSql.s.c(String A_0)
at Devart.Data.PostgreSql.s.g(af A_0)
at Devart.Data.PostgreSql.s.e(af A_0)
at Devart.Data.PostgreSql.af.ah()
at Devart.Data.PostgreSql.PgSqlCommand.InternalPrepare(Boolean implicitPrepare, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at Devart.Common.DbCommandBase.ExecuteNonQuery()
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25

Best regards,
Anders

[Shalex]

Thank you for your report. We have reproduced System.OutOfMemoryException when inserting 200+ MB file into a bytea column using EF v4. We will investigate the issue and post here about the results.

[asj]

Thank you for your report. We have reproduced System.OutOfMemoryException when inserting 200+ MB file into a bytea column using EF v4. We will investigate the issue and post here about the results.

Hello Shalex,

Any news of when a fix will be released?

Best regards,
Anders

[Shalex]

The reasons of the issue.

1. The Entity Framework side.
Additional memory costs by Entity Framework application are caused by change tracking, numerous additional ORM structures in the memory, and usage of standard code in the setter of binary property:

Code: Select all

_Byteacolumn = StructuralObject.SetValidValue(value, true);

The SetValidValue code clones array of bytes in its implementation, which leads to the additional memory costs. We do not know the reasons why Microsoft chose such behaviour of StructuralObject.SetValidValue for the byte array.

2. The ADO.NET side.
Currently, our ADO.NET level increases an amount of used memory in several times comparing to the size of actual bytea data. We are investigating the possibility to change the approach of handling such situation.

[Shalex]

The ADO.NET side of the problem is fixed. We will post here when the corresponding build of dotConnect for PostgreSQL is available for download.

[curelom]

I get the same error in Oracle. I imagine this problem extends to all the databases.

[Shalex]

I get the same error in Oracle. I imagine this problem extends to all the databases.

Thank you for your report. We are investigating the issue.

[muthu]

I am having the same issue using PostgreSQL LinqConnect..

[MariiaI]

I am having the same issue using PostgreSQL LinqConnect..

Thank you for the report. We have reproduced this issue with LinqConnect. We will investigate it and inform you about the results as soon as possible.

[muthu]

any news on the bug fix ?

[Shalex]

The investigation is in the progress. We will keep you updated concerning the status of the corresponding request.

[Keenan]

I encounter a similar problem inserting large binary files into a mysql database through EF4(using devart's dotConnect for MySQL).
I get the exception and message below when inserting mp4 video files, approx. 4mb in size.

System.InvalidOperationException
Message: "Connection must be opened."

StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.MySql.MySqlConnection.Rollback()
at Devart.Data.MySql.MySqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Entity.Internal.InternalContext.SaveChanges()
at System.Data.Entity.Internal.LazyInternalContext.SaveChanges()
at System.Data.Entity.DbContext.SaveChanges()

I've found that if I only import the binary objects for pdfs of similar size(~2.5mb) that everything works well.

Is this still an open problem for dotConnect for MySQL?

Thank you,
Keenan

[Shalex]

Does the issue persist with the latest (7.7.301) build of dotConnect for MySQL? Please send us a small test project with the corresponding DDL/DML script so that we can reproduce the issue in our environment.

[Keenan]

The upload issue was resolved by increasing the max_allowed_packet to 50MB in mysql. Although when attempting to retrieve any file over 10MB I get the following error:

System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode

StackTrace:
at Devart.Data.MySql.bl.b(Boolean A_0)
at Devart.Data.MySql.bl.n()
at Devart.Data.MySql.bu.m()
at Devart.Data.MySql.af.b(ab[] A_0, Int32 A_1)
at Devart.Data.MySql.af.a(Byte[] A_0, Int32 A_1, Boolean A_2)
at Devart.Data.MySql.i.b()
at Devart.Data.MySql.i.v()
at Devart.Data.MySql.MySqlCommand.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at Devart.Data.MySql.Entity.x.a(CommandBehavior A_0)
at Devart.Common.Entity.eb.b(CommandBehavior A_0)
at System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)


Here's the EF4 code that throws the error:

using (SalesModelDB db = new SalesModelDB())
{
fileData = db.DataDictionaries
.Where(d => d.Id == fileId)
.Select(f => f.FileData)
.FirstOrDefault();
}

Is there a way to increase the fetch limit or switch the mode from fetchall to something else to allow me to get these larger binary files?

Thank you,
Keenan

[MariiaI]

Although when attempting to retrieve any file over 10MB I get the following error:
System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode

Thank you for the report. We have reproduced this issue. We will investigate it and inform you about the results as soon as possible.