Deadlock on Devart.Common.DbConnectionPool.GetObject

Discussion of open issues, suggestions and bugs regarding ADO.NET provider for MySQL
Post Reply
tmezzena
Posts: 2
Joined: Wed 09 May 2018 21:54

Deadlock on Devart.Common.DbConnectionPool.GetObject

Post by tmezzena » Thu 10 May 2018 08:57

Hi,

I had been using DevArt dotConnect for MySql 8.6.729.0, without connection pool, on our site for long time but, because the user growth and changes some parts to async logic (more connections), I got the Windows limitation for connections (https://dev.mysql.com/doc/refman/5.5/en ... on-windows).
So, I changed the connection string to use connection pool.
It worked but I got to experience other problem: After 6 or 8 hours the web site freezes and I have to change to a new application pool to run the site again.
The site is fast all the time but only few minutes before the freeze, the site get slow.
I used the tool "Debug Diagnostic Tool" to get information about the period near the freeze, and I got that 32% of app pool threads are locked in Devart.Common.DbConnectionPool.GetObject().
I renewed the licence of dotConnect fo MySql 8.10.1134.0 and deployed this new version but I've been gotten the same problem.
I saw other customers that had similar problems but only one (viewtopic.php?f=1&t=35462&sid=d09962e2a ... 9&start=30) was detected a problem and fixed; but it was dotConnect for Oracle and I coudn't see the fix on change history.
I have the "DebugDiag Analysis Report" to send you and I coudn't reproduce this problem on a simple sample.

Do you have some help/tips for this problem?

Thanks,

Marco.

----

Folow the stack trace for the lock:

Thread 58 - System ID 5840

Entry point clr!Thread::intermediateThreadProc
Create time 09/05/2018 14:21:42
Time spent in user mode 0 Days 00:00:36.937
Time spent in kernel mode 0 Days 00:00:01.703

This thread is waiting to enter a .NET Lock

Code: Select all


.NET Call Stack

[[GCFrame]] 
[[HelperMethodFrame] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object) 
Devart.Common.DbConnectionPool.GetObject(Devart.Common.DbConnectionBase)+2e 
Devart.Common.DbConnectionFactory.b(Devart.Common.DbConnectionBase)+d0 
Devart.Common.DbConnectionClosed.Open(Devart.Common.DbConnectionBase)+60 
Devart.Common.DbConnectionBase.Open()+138 
Devart.Data.MySql.MySqlConnection.m()+94 
Devart.Data.MySql.MySqlConnection.Open()+c7 
Dapper.SqlMapper+d__136`1[[System.__Canon, mscorlib]].MoveNext()+119 
System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1)+1dd 
System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1)+37 
Dapper.SqlMapper.Query[[System.__Canon, mscorlib]](System.Data.IDbConnection, System.String, System.Object, System.Data.IDbTransaction, Boolean, System.Nullable`1, System.Nullable`1)+17b 
VeloxTicketsV3.Repository.PartnerRepositoryReadOnly.Get(System.String)+e3 
VeloxTicketsV3.Application.AppService.PartnerService.Get(System.String)+3c 
VeloxTicketsV3.Web.Common.ServiceHelper.FactoryHelper.GetPartnerByCode(System.String)+3c 
VeloxTicketsV3.Web.Common.Infra.ContextLoadData.GetPartnerByCode(System.String)+2b 
VeloxTicketsV3.Web.Common.Infra.ContextLoadData.Load(System.Web.HttpContextBase, System.Web.Routing.RouteData)+a32 
VeloxTicketsV3.Web.Common.Infra.ContextRequestBinder.BindModel(System.Web.Mvc.ControllerContext, System.Web.Mvc.ModelBindingContext)+f7 
System.Web.Mvc.ControllerActionInvoker.GetParameterValue(System.Web.Mvc.ControllerContext, System.Web.Mvc.ParameterDescriptor)+159 
System.Web.Mvc.ControllerActionInvoker.GetParameterValues(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor)+6a 
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass3_1.b__0(System.AsyncCallback, System.Object)+1d6 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1[[System.Boolean, mscorlib]].CallBeginDelegate(System.AsyncCallback, System.Object)+f 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Boolean, mscorlib]].Begin(System.AsyncCallback, System.Object, Int32)+82 
System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(System.Web.Mvc.ControllerContext, System.String, System.AsyncCallback, System.Object)+14c 
System.Web.Mvc.Controller+<>c.b__152_0(System.AsyncCallback, System.Object, ExecuteCoreState)+1c 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.Controller+ExecuteCoreState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+1f 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82 
System.Web.Mvc.Controller.BeginExecuteCore(System.AsyncCallback, System.Object)+1ae 
System.Web.Mvc.Controller+<>c.b__151_1(System.AsyncCallback, System.Object, System.Web.Mvc.Controller)+11 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.__Canon, mscorlib]].CallBeginDelegate(System.AsyncCallback, System.Object)+15 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82 
System.Web.Mvc.Controller.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+175 
System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+11 
System.Web.Mvc.MvcHandler+<>c.b__20_0(System.AsyncCallback, System.Object, ProcessRequestState)+24 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.MvcHandler+ProcessRequestState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+1f 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82 
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object)+190 
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+62 
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+4d 
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+182 
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+9e 
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+619 
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+97 
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1ba 
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3e4 
[[InlinedCallFrame]] 
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+20 
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+60 
[[InlinedCallFrame]] 
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+52c 
[[InlinedCallFrame]] 
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+20 
[[ContextTransitionFrame]] 

Full Call Stack

ntdll!NtWaitForMultipleObjects+c    
KERNELBASE!WaitForMultipleObjectsEx+cc    
clr!WaitForMultipleObjectsEx_SO_TOLERANT+3c    
clr!Thread::DoAppropriateWaitWorker+237    
clr!Thread::DoAppropriateWait+64    
clr!CLREventBase::WaitEx+128    
clr!CLREventBase::Wait+1a    
clr!AwareLock::EnterEpilogHelper+a8    
[[GCFrame]]    
clr!AwareLock::EnterEpilog+48    
clr!AwareLock::Enter+4a    
clr!JITutil_MonContention+6a    
[[HelperMethodFrame] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object)    
Devart.Common.DbConnectionPool.GetObject(Devart.Common.DbConnectionBase)+2e    
Devart.Common.DbConnectionFactory.b(Devart.Common.DbConnectionBase)+d0    
Devart.Common.DbConnectionClosed.Open(Devart.Common.DbConnectionBase)+60    
Devart.Common.DbConnectionBase.Open()+138    
Devart.Data.MySql.MySqlConnection.m()+94    
Devart.Data.MySql.MySqlConnection.Open()+c7    
Dapper.SqlMapper+<QueryImpl>d__136`1[[System.__Canon, mscorlib]].MoveNext()+119    
System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)+1dd    
System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)+37    
Dapper.SqlMapper.Query[[System.__Canon, mscorlib]](System.Data.IDbConnection, System.String, System.Object, System.Data.IDbTransaction, Boolean, System.Nullable`1<Int32>, System.Nullable`1<System.Data.CommandType>)+17b    
VeloxTicketsV3.Repository.PartnerRepositoryReadOnly.Get(System.String)+e3    
VeloxTicketsV3.Repository.PartnerRepositoryReadOnly.Get(System.String)+e3    
VeloxTicketsV3.Application.AppService.PartnerService.Get(System.String)+3c    
VeloxTicketsV3.Web.Common.ServiceHelper.FactoryHelper.GetPartnerByCode(System.String)+3c    
VeloxTicketsV3.Web.Common.Infra.ContextLoadData.GetPartnerByCode(System.String)+2b    
VeloxTicketsV3.Web.Common.Infra.ContextLoadData.Load(System.Web.HttpContextBase, System.Web.Routing.RouteData)+a32    
VeloxTicketsV3.Web.Common.Infra.ContextRequestBinder.BindModel(System.Web.Mvc.ControllerContext, System.Web.Mvc.ModelBindingContext)+f7    
System.Web.Mvc.ControllerActionInvoker.GetParameterValue(System.Web.Mvc.ControllerContext, System.Web.Mvc.ParameterDescriptor)+159    
System.Web.Mvc.ControllerActionInvoker.GetParameterValue(System.Web.Mvc.ControllerContext, System.Web.Mvc.ParameterDescriptor)+159    
System.Web.Mvc.ControllerActionInvoker.GetParameterValues(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor)+6a    
System.Web.Mvc.ControllerActionInvoker.GetParameterValues(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor)+6a    
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass3_1.<BeginInvokeAction>b__0(System.AsyncCallback, System.Object)+1d6    
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass3_1.<BeginInvokeAction>b__0(System.AsyncCallback, System.Object)+1d6    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1[[System.Boolean, mscorlib]].CallBeginDelegate(System.AsyncCallback, System.Object)+f    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1[[System.Boolean, mscorlib]].CallBeginDelegate(System.AsyncCallback, System.Object)+f    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Boolean, mscorlib]].Begin(System.AsyncCallback, System.Object, Int32)+82    
System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(System.Web.Mvc.ControllerContext, System.String, System.AsyncCallback, System.Object)+14c    
System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(System.Web.Mvc.ControllerContext, System.String, System.AsyncCallback, System.Object)+14c    
System.Web.Mvc.Controller+<>c.<BeginExecuteCore>b__152_0(System.AsyncCallback, System.Object, ExecuteCoreState)+1c    
System.Web.Mvc.Controller+<>c.<BeginExecuteCore>b__152_0(System.AsyncCallback, System.Object, ExecuteCoreState)+1c    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.Controller+ExecuteCoreState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+1f    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.Controller+ExecuteCoreState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+1f    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82    
System.Web.Mvc.Controller.BeginExecuteCore(System.AsyncCallback, System.Object)+1ae    
System.Web.Mvc.Controller.BeginExecuteCore(System.AsyncCallback, System.Object)+1ae    
System.Web.Mvc.Controller+<>c.<BeginExecute>b__151_1(System.AsyncCallback, System.Object, System.Web.Mvc.Controller)+11    
System.Web.Mvc.Controller+<>c.<BeginExecute>b__151_1(System.AsyncCallback, System.Object, System.Web.Mvc.Controller)+11    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.__Canon, mscorlib]].CallBeginDelegate(System.AsyncCallback, System.Object)+15    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.__Canon, mscorlib]].CallBeginDelegate(System.AsyncCallback, System.Object)+15    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82    
System.Web.Mvc.Controller.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+175    
System.Web.Mvc.Controller.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+175    
System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+11    
System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+11    
System.Web.Mvc.MvcHandler+<>c.<BeginProcessRequest>b__20_0(System.AsyncCallback, System.Object, ProcessRequestState)+24    
System.Web.Mvc.MvcHandler+<>c.<BeginProcessRequest>b__20_0(System.AsyncCallback, System.Object, ProcessRequestState)+24    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.MvcHandler+ProcessRequestState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+1f    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.MvcHandler+ProcessRequestState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+1f    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82    
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+82    
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object)+190    
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object)+190    
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+62    
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+62    
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+4d    
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)+4d    
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+182    
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+182    
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+9e    
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+619    
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+619    
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+97    
clr!JIT_Security_Prolog_Framed+15c    
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1ba    
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+1ba    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3e4    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+3e4    
[[InlinedCallFrame]]    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+20    
webengine4!W3_MGD_HANDLER::ProcessNotification+62    
webengine4!W3_MGD_HANDLER::DoWork+32a    
webengine4!RequestDoWork+39f    
webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+18    
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+2dd    
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+4af    
iiscore!NOTIFICATION_CONTEXT::CallModules+2b    
iiscore!NOTIFICATION_MAIN::DoStateRequestExecuteHandler+40    
iiscore!NOTIFICATION_MAIN::DoWork+11e    
iiscore!W3_CONTEXT_BASE::ContinueNotificationLoop+32    
iiscore!W3_CONTEXT_BASE::IndicateCompletion+a0    
webengine4!W3_MGD_HANDLER::IndicateCompletion+45    
webengine4!MgdIndicateCompletion+22    
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+60    
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+60    
[[InlinedCallFrame]]    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+52c    
[[InlinedCallFrame]]    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+20    
clr!UM2MThunk_WrapperHelper+10    
clr!UM2MThunk_Wrapper+76    
clr!Thread::DoADCallBack+bc    
[[ContextTransitionFrame]]    
clr!UM2MDoADCallBack+92    
0x017ce967    
webengine4!W3_MGD_HANDLER::ProcessNotification+62    
webengine4!ProcessNotificationCallback+33    
clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+1d6    
clr!ThreadpoolMgr::ExecuteWorkRequest+4f    
clr!Thread::intermediateThreadProc+55    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b 


tmezzena
Posts: 2
Joined: Wed 09 May 2018 21:54

Re: Deadlock on Devart.Common.DbConnectionPool.GetObject

Post by tmezzena » Tue 15 May 2018 10:01

Hello! I'm waiting some iteration...

Pinturiccio
Devart Team
Posts: 2138
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock on Devart.Common.DbConnectionPool.GetObject

Post by Pinturiccio » Wed 16 May 2018 09:46

Sorry for the late response.
tmezzena wrote:I renewed the licence of dotConnect fo MySql 8.10.1134.0 and deployed this new version but I've been gotten the same problem.
I saw other customers that had similar problems but only one (viewtopic.php?f=1&t=35462&sid=d09962e2a ... 9&start=30) was detected a problem and fixed; but it was dotConnect for Oracle and I coudn't see the fix on change history.
The changes were made on the base level, and they are common for all the providers, including dotConnect for MySQL - not just dotConnect for Oracle. These changes are included into dotConnect for MySQL 8.10.1134. The user from the topic above tested these changes, and they helped him.

Please make sure that the application loads new Devart assembly versions at run-time, and not the old ones, cached somewhere. If you make sure of it, and the issue is still reproduced with the latest dotConnect for MySQL version, let us know about it.

Post Reply