NullReferenceException on SubmitChanges

[msw10100]

Using Postgres 9.2.2 with LinqConnect 4.4.383.0, I am experiencing a reproducible 'NullReferenceException' crash after doing a series of InsertOnSubmit() operations. I've confirmed that my code is not passing in bogus information, and I've recreated the problem in an isolated standalone dummy application.

Here is the exception:

Code: Select all

System.NullReferenceException was unhandled
  HResult=-2147467261
  Message=Object reference not set to an instance of an object.
  Source=Devart.Data.Linq
  StackTrace:
       at Devart.Data.Linq.Engine.c5.h(IObjectEntry A_0)
       at Devart.Data.Linq.Engine.c5.j()
       at Devart.Data.Linq.Engine.c5.b.b()
       at Devart.Data.Linq.Engine.b5.a(ConflictMode A_0)
       at Devart.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
       at Devart.Data.Linq.DataContext.SubmitChanges()
       at RapidWriteTest.Program.Main(String[] args) in c:\Users\mwatson\Documents\Visual Studio 2012\Projects\RapidWriteTest\RapidWriteTest\Program.cs:line 56
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: 

Here is the code that is running:

Code: Select all

    var connString = "User Id=postgres;Password=XXXXXXX;Host=localhost;Database=blueridge;Persist Security Info=True;Initial Schema=public";
    const int tagsToWrite = 100000000;
    var context = new RapidWriteFailContext(connString);
    for (var i = 0; i < tagsToWrite; i++)
    {
            var update = new TagUpdate
                {
                    Datatype = 1,
                    TagPathId = tagPathId, // the record ID of a record in the tag_path table
                    TimestampedAt = DateTime.Now,
                    Value = Membership.GeneratePassword(32, 8) // this just generates a random string to use in populating the field
                };
            context.TagUpdates.InsertOnSubmit(update);
        if (i > 0 && i%200 == 0)
        {
            context.SubmitChanges();
            context.Dispose();
            context = null;
            context = new RapidWriteFailContext(connString);
        }
                
        if (i > 0 && i%1000 == 0)
        {
            Console.WriteLine("...{0}/{1} written", i, tagsToWrite);
        }
    }

The database I'm using has tables created by the SQL below, and I created the DataContext from the database, so I know it should match the underlying data:

Code: Select all

CREATE TABLE tag_path
(
    id serial NOT NULL,
    path varchar(200),
    CONSTRAINT tag_path_id PRIMARY KEY (id)
);

CREATE INDEX tag_path_path ON tag_path USING btree (path);

CREATE TABLE tag_path_property
(
    id serial NOT NULL,
    tag_path_id int,
    key varchar(200),
    value varchar(200),
    CONSTRAINT tag_path_property_id PRIMARY KEY (id)    ,
    CONSTRAINT tag_path_property_tag_path_fk FOREIGN KEY (tag_path_id)
        REFERENCES tag_path (id) MATCH SIMPLE
        ON UPDATE NO ACTION ON DELETE NO ACTION
);

CREATE INDEX tag_path_property_key ON tag_path_property USING btree (key);
CREATE INDEX tag_path_property_value ON tag_path_property USING btree (value);

CREATE INDEX tag_path_property_tag_path_id ON tag_path_property USING btree (tag_path_id);

CREATE TABLE tag_update
(
    id serial NOT NULL,
    tag_path_id int,
    timestamped_at timestamp NOT NULL,
    datatype int NOT NULL,
    value text,
    CONSTRAINT tag_update_id PRIMARY KEY (id)    ,
    CONSTRAINT tag_update_tag_path_fk FOREIGN KEY (tag_path_id)
        REFERENCES tag_path (id) MATCH SIMPLE
        ON UPDATE NO ACTION ON DELETE NO ACTION
);

CREATE INDEX tag_update_timestamped_at ON tag_update USING btree (timestamped_at);
CREATE INDEX tag_update_datatype ON tag_update USING btree (datatype);

CREATE INDEX tag_update_tag_path_id ON tag_update USING btree (tag_path_id);

CREATE TABLE tag_tree
(
    id serial NOT NULL,
    tag_tree_id int,
    name varchar(200),
    CONSTRAINT tag_tree_id PRIMARY KEY (id)    ,
    CONSTRAINT tag_tree_tag_tree_fk FOREIGN KEY (tag_tree_id)
        REFERENCES tag_tree (id) MATCH SIMPLE
        ON UPDATE NO ACTION ON DELETE NO ACTION
);

CREATE INDEX tag_tree_name ON tag_tree USING btree (name);

CREATE INDEX tag_tree_tag_tree_id ON tag_tree USING btree (tag_tree_id);

[msw10100]

For a temporary workaround, I trap the exception and retry the whole set of operations. It usually works on the second attempt, but not always. This will not be acceptable for production, though, because in the real program (as opposed to the above standalone dummy program) the context may get submitted by other operations that occur, meaning that it would submit some changes multiple times.

[msw10100]

I tried shifting to adding the new records to a List<TagUpdate> and then using context.InsertAllOnSubmit(), and the exception still occurs.

[msw10100]

I updated the code in the message above to show that I'm writing a ton of records:

const int tagsToWrite = 100000000;

In the real application, it is writing nearly continuously for stretches at a time, and we may get millions of records written in a day. Many of the records are written in bursts of 60-100 inserts at a time within a single context. In the example code, I use 200 as the number of records to write at a time. The crash occurs pretty reliably within the first few hundred thousand records, even if we write just 3-5 records at a time.

[MariiaI]

Probably, the error occurs because of this code:

Code: Select all

context.Dispose();
context = null;

According to your scenario there is no need in disposing DataContext object for this unit of work. You could use one DataContext to insert new objects to the necessary database table. Please try this code and tell us about the results:

Code: Select all

using (var context = new RapidWriteFailContext())
            {
                List<TagUpdate> list = new List<TagUpdate>();
                for (var i = 0; i < tagsToWrite; i++)
                {
                    var update = new TagUpdate
                    {
                        Datatype = 1,
                        TagPathId = tagPathId, 
                        TimestampedAt = DateTime.Now,
                        Value = Membership.GeneratePassword(32, 8) 
                    };
                    list.Add(update);                    
                }
                context.TagUpdates.InsertAllOnSubmit(list);
                context.SubmitChanges();          
            }

If this doesn't help, please send us a small test project so that we are able to find a more suitable solution for you.

[msw10100]

I will try it, but the reason the code does that is because the real program effectively works that way. The real program receives a message from a message bus, creates a DataContext, does some work with the message, which involves inserting records contained within the message, then is done, so discards the DataContext.

This is in line with other recommendations made on this forum that DataContexts are lightweight objects that should be created, used, then destroyed for granular work. And that using one for inserts continuously is problematic because too many objects become attached to the DataContext, and slow it down over time, as in the reply to this post:

http://forums.devart.com/viewtopic.php? ... ght#p95543

If creating and destroying a DataContext results in a crash, I suspect there's something else wrong. also, as the crash does not happen on the dispose but on the submit changes,it seems unlikely to be related to the dispose operation directly.

I will make the test and report back.

[MariiaI]

Thank you for the sample project. We have reproduced this issue, it seems to be related to the entity cache. We will investigate it and inform you about the results as soon as possible.
Please try setting

Code: Select all

context.EntityCachingMode = EntityCachingMode.StrongReference;

to avoid the error.
Please tell us if this helps.

[msw10100]

The StrongRefence setting does seem to prevent the crash, yes.

Cheers!

[MariiaI]

We have reproduced this issue, it seems to be related to the entity cache. We will investigate it and inform you about the results as soon as possible.

The issue has been fixed. We will inform you when the correspoding build of LinqConnect is available for download.

[MariiaI]

New build of LinqConnect 4.4.403 is available for download now!
It can be downloaded from http://www.devart.com/linqconnect/download.html (trial version) or from Registered Users' Area (for users with active subscription only).
For more information, please refer to http://forums.devart.com/viewtopic.php?f=31&t=28602.