Error when inserting large binary files using EF4

Error when inserting large binary files using EF4

Postby asj » Mon 06 Feb 2012 11:16

Hello Devart,

I am experiencing problems when inserting binary files into a postgres bytea column using Entity Framework 4 and dotConnect. When inserting a file i get either a System.InvalidOperationException("Connection must be opened.") (150 MB file) or a System.OutOfMemoryException (200+ MB file).

I have a test project available demonstrating the behavior. Do you have a place where i may upload it?


Full tracebacks for the Exceptions are as follows:
System.InvalidOperationException
Message: "Connection must be opened."

StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25


System.OutOfMemoryException:
Message: "Exception of type 'System.OutOfMemoryException' was thrown."

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Devart.Data.PostgreSql.o.a(Byte[] A_0, Int32 A_1, Int32 A_2, Int32 A_3)
at Devart.Data.PostgreSql.o.b(Byte[] A_0, Int32 A_1, Int32 A_2)
at Devart.Data.PostgreSql.o.a(Char A_0)
at Devart.Data.PostgreSql.s.b(Char A_0)
at Devart.Data.PostgreSql.s.c(String A_0)
at Devart.Data.PostgreSql.s.g(af A_0)
at Devart.Data.PostgreSql.s.e(af A_0)
at Devart.Data.PostgreSql.af.ah()
at Devart.Data.PostgreSql.PgSqlCommand.InternalPrepare(Boolean implicitPrepare, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at Devart.Common.DbCommandBase.ExecuteNonQuery()
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25

Best regards,
Anders
asj
 
Posts: 8
Joined: Fri 16 Sep 2011 08:05

Postby Shalex » Tue 07 Feb 2012 14:39

Thank you for your report. We have reproduced System.OutOfMemoryException when inserting 200+ MB file into a bytea column using EF v4. We will investigate the issue and post here about the results.
Shalex
Devart Team
 
Posts: 7775
Joined: Thu 14 Aug 2008 12:44

Postby asj » Wed 28 Mar 2012 12:31

Shalex wrote:Thank you for your report. We have reproduced System.OutOfMemoryException when inserting 200+ MB file into a bytea column using EF v4. We will investigate the issue and post here about the results.


Hello Shalex,

Any news of when a fix will be released?

Best regards,
Anders
asj
 
Posts: 8
Joined: Fri 16 Sep 2011 08:05

Postby Shalex » Mon 02 Apr 2012 12:39

The reasons of the issue.

1. The Entity Framework side.
Additional memory costs by Entity Framework application are caused by change tracking, numerous additional ORM structures in the memory, and usage of standard code in the setter of binary property:
Code: Select all
_Byteacolumn = StructuralObject.SetValidValue(value, true);

The SetValidValue code clones array of bytes in its implementation, which leads to the additional memory costs. We do not know the reasons why Microsoft chose such behaviour of StructuralObject.SetValidValue for the byte array.

2. The ADO.NET side.
Currently, our ADO.NET level increases an amount of used memory in several times comparing to the size of actual bytea data. We are investigating the possibility to change the approach of handling such situation.
Shalex
Devart Team
 
Posts: 7775
Joined: Thu 14 Aug 2008 12:44

Postby Shalex » Mon 02 Apr 2012 15:20

The ADO.NET side of the problem is fixed. We will post here when the corresponding build of dotConnect for PostgreSQL is available for download.
Shalex
Devart Team
 
Posts: 7775
Joined: Thu 14 Aug 2008 12:44

Re: Error when inserting large binary files using EF4

Postby curelom » Fri 27 Apr 2012 20:39

I get the same error in Oracle. I imagine this problem extends to all the databases.
curelom
 
Posts: 19
Joined: Wed 17 Feb 2010 21:10

Re: Error when inserting large binary files using EF4

Postby Shalex » Fri 04 May 2012 16:39

curelom wrote:I get the same error in Oracle. I imagine this problem extends to all the databases.

Thank you for your report. We are investigating the issue.
Shalex
Devart Team
 
Posts: 7775
Joined: Thu 14 Aug 2008 12:44

Re: Error when inserting large binary files using EF4

Postby muthu » Thu 05 Jul 2012 11:47

I am having the same issue using PostgreSQL LinqConnect..
muthu
 
Posts: 10
Joined: Tue 28 Sep 2010 19:31

Re: Error when inserting large binary files using EF4

Postby MariiaI » Fri 06 Jul 2012 08:08

muthu wrote:I am having the same issue using PostgreSQL LinqConnect..

Thank you for the report. We have reproduced this issue with LinqConnect. We will investigate it and inform you about the results as soon as possible.
MariiaI
Devart Team
 
Posts: 1472
Joined: Mon 13 Feb 2012 08:17

Re: Error when inserting large binary files using EF4

Postby muthu » Wed 08 Aug 2012 12:24

any news on the bug fix ?
muthu
 
Posts: 10
Joined: Tue 28 Sep 2010 19:31

Re: Error when inserting large binary files using EF4

Postby Shalex » Fri 10 Aug 2012 11:30

The investigation is in the progress. We will keep you updated concerning the status of the corresponding request.
Shalex
Devart Team
 
Posts: 7775
Joined: Thu 14 Aug 2008 12:44

Re: Error when inserting large binary files using EF4

Postby Keenan » Wed 28 Aug 2013 18:50

I encounter a similar problem inserting large binary files into a mysql database through EF4(using devart's dotConnect for MySQL).
I get the exception and message below when inserting mp4 video files, approx. 4mb in size.

System.InvalidOperationException
Message: "Connection must be opened."

StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.MySql.MySqlConnection.Rollback()
at Devart.Data.MySql.MySqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Entity.Internal.InternalContext.SaveChanges()
at System.Data.Entity.Internal.LazyInternalContext.SaveChanges()
at System.Data.Entity.DbContext.SaveChanges()

I've found that if I only import the binary objects for pdfs of similar size(~2.5mb) that everything works well.

Is this still an open problem for dotConnect for MySQL?

Thank you,
Keenan
Keenan
 
Posts: 3
Joined: Wed 28 Aug 2013 18:36

Re: Error when inserting large binary files using EF4

Postby Shalex » Mon 02 Sep 2013 16:08

Does the issue persist with the latest (7.7.301) build of dotConnect for MySQL? Please send us a small test project with the corresponding DDL/DML script so that we can reproduce the issue in our environment.
Shalex
Devart Team
 
Posts: 7775
Joined: Thu 14 Aug 2008 12:44

Re: Error when inserting large binary files using EF4

Postby Keenan » Mon 16 Sep 2013 17:59

The upload issue was resolved by increasing the max_allowed_packet to 50MB in mysql. Although when attempting to retrieve any file over 10MB I get the following error:

System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode

StackTrace:
at Devart.Data.MySql.bl.b(Boolean A_0)
at Devart.Data.MySql.bl.n()
at Devart.Data.MySql.bu.m()
at Devart.Data.MySql.af.b(ab[] A_0, Int32 A_1)
at Devart.Data.MySql.af.a(Byte[] A_0, Int32 A_1, Boolean A_2)
at Devart.Data.MySql.i.b()
at Devart.Data.MySql.i.v()
at Devart.Data.MySql.MySqlCommand.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at Devart.Data.MySql.Entity.x.a(CommandBehavior A_0)
at Devart.Common.Entity.eb.b(CommandBehavior A_0)
at System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)


Here's the EF4 code that throws the error:

using (SalesModelDB db = new SalesModelDB())
{
fileData = db.DataDictionaries
.Where(d => d.Id == fileId)
.Select(f => f.FileData)
.FirstOrDefault();
}

Is there a way to increase the fetch limit or switch the mode from fetchall to something else to allow me to get these larger binary files?

Thank you,
Keenan
Keenan
 
Posts: 3
Joined: Wed 28 Aug 2013 18:36

Re: Error when inserting large binary files using EF4

Postby MariiaI » Tue 17 Sep 2013 12:16

Keenan wrote: Although when attempting to retrieve any file over 10MB I get the following error:
System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode

Thank you for the report. We have reproduced this issue. We will investigate it and inform you about the results as soon as possible.
MariiaI
Devart Team
 
Posts: 1472
Joined: Mon 13 Feb 2012 08:17

Next

Return to dotConnect for PostgreSQL