Permanent failure if First Query attempted while DB server is unavailable

Discussion of open issues, suggestions and bugs regarding Entity Framework support in ADO.NET Data providers
Post Reply
PVHills
Posts: 3
Joined: Wed 14 Apr 2021 21:43

Permanent failure if First Query attempted while DB server is unavailable

Post by PVHills » Wed 14 Apr 2021 22:24

Hi There,
We have several microservices built on top of aspnetcore and entity framework and using the MySQL EntityFramework connector from Devart. I've noticed that if the first query that any of our applications attempts fails due to the data store being inaccessible, then the failure seems to be cached, and when the DB is finally alive, the exception is still returned to the user code.

As part of the diagnosis of this problem I tried disabling the database when the application is running and in this situation the application appears to be able to recover once the DB is available again, so I suspect the problem is related to a connection that may be cached during model initialisation.

I've also tried disabling the service provider caching and this also lets the application recover but at the cost of having to initialise the service provider each time, so this can't be used in our application but lends weight to the idea that the error is related to some form of caching.

Has anyone else had/seen this problem before? What was your solution?
Here is a stack trace for context

Code: Select all

   at   .    (String , String , String , String , Int32 , String , Int32 , SshOptions  , SslOptions  , ProxyOptions  , MySqlHttpOptions  , HttpOptions  , Int32  )
   at   .(MySqlConnection , String , String , String , String , Int32 , Int32 , MySqlProtocol  , Boolean  , Boolean  )
   at   ..ctor(   , MySqlConnection )
   at   .   (  , Object , DbConnectionBase )
   at  .(  ,   , DbConnectionBase )
   at  .(  , DbConnectionBase )
   at  .(DbConnectionBase )
   at  .(DbConnectionBase )
   at  .(DbConnectionBase )
   at  .   (DbConnectionBase )
   at Devart.Common.DbConnectionBase.Open()
   at Devart.Data.MySql.MySqlConnection.()
   at Devart.Data.MySql.MySqlConnection.Open()
   at Devart.Common.Entity.c7.a(DbConnection A_0)
   at Devart.Common.Entity.c7.b(DbConnection A_0)
   at Devart.Data.MySql.Entity.ad.a(DbConnection A_0)
   at Devart.Data.MySql.Entity.aa.b(RelationalOptionsExtension A_0)
   at Devart.Common.Entity.c3.a()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Devart.Common.Entity.c3.b()
   at Devart.Data.MySql.Entity.w..ctor(TypeMappingSourceDependencies A_0, RelationalTypeMappingSourceDependencies A_1, c4 A_2)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkRelationalServicesBuilder.<>c.<TryAddCoreServices>b__3_0(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__7_3(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
   at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
   at Microsoft.EntityFrameworkCore.DbContext.get_Model()
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.get_EntityType()
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.CheckState()
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.get_EntityQueryable()
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.System.Linq.IQueryable.get_Provider()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.Include[TEntity,TProperty](IQueryable`1 source, Expression`1 navigationPropertyPath)
(Our stack from here on)

Shalex
Site Admin
Posts: 9275
Joined: Thu 14 Aug 2008 12:44

Re: Permanent failure if First Query attempted while DB server is unavailable

Post by Shalex » Thu 15 Apr 2021 17:08

Please try turning off pooling ("Pooling=false;" in the connection string) or clear the pool explicitly with MySqlConnection.ClearAllPools(true).

For more information, refer to:
* https://www.devart.com/dotconnect/mysql ... tring.html
* https://www.devart.com/dotconnect/mysql ... lean).html

PVHills
Posts: 3
Joined: Wed 14 Apr 2021 21:43

Re: Permanent failure if First Query attempted while DB server is unavailable

Post by PVHills » Thu 15 Apr 2021 20:52

Hi thanks for your response. I tried changing this config using Pooling=false; in the connection string. This didn't help. My casual model of the breakage is that the model builder is failing during initialisation, and the exception is being captured and passed on to the user code. System.Lazy is well known for this behaviour and I noticed that the failing call stack contains a use of one of these. I suspect that the connection isn't even retried, just that the failed lazy evaluation is played back

Code: Select all

   at Devart.Common.Entity.c3.a()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Devart.Common.Entity.c3.b()
I've managed to work around this by supplying an open DB connection in the configuration options action, here's my workaround:

Code: Select all

            services.AddTransient<DbConnection, MySqlConnection>();
            services.AddDbContext<BusinessContext>((sp, opt) =>
            {
                lock (this)
                {
                    var sql = sp.GetRequiredService<DbConnection>();
                    sql.ConnectionString = Configuration.GetConnectionString("connectionstring");
                    if (!_isConnected)
                    {
                        sql.Open(); // Ensure we can actually get a connection
                        _isConnected = true;
                    }
                    opt.UseMySql(sql);
                }
            });
This throws an exception if the first connection attempt occurs when the DB is unavailable, and prevents the devart component from attempting the model creation until the first connection succeeds. This isn't great, and I'd prefer not to have to do this, and it won't guarantee success

Shalex
Site Admin
Posts: 9275
Joined: Thu 14 Aug 2008 12:44

Re: Permanent failure if First Query attempted while DB server is unavailable

Post by Shalex » Tue 20 Apr 2021 14:19

The bug with connection recovery after the initial connection attempt failure in EF Core is fixed. We will notify you when a new public build of dotConnect for MySQL is available for download.

PVHills
Posts: 3
Joined: Wed 14 Apr 2021 21:43

Re: Permanent failure if First Query attempted while DB server is unavailable

Post by PVHills » Tue 20 Apr 2021 20:50

Great news! Looking forward to installing it

Post Reply