Devart Forums

Discussion forums for open issues and questions concerning database tools, data access components and developer tools from Devart
https://forums.devart.com/

Deadlock in ConnectionPool

https://forums.devart.com/viewtopic.php?f=1&t=33352

Page 1 of 1

Deadlock in ConnectionPool

Posted: Thu 10 Mar 2016 16:40
by BGrojer
Hi,
we have been encountering regular ConnectionPool deadlocks under heavy load.

Inspecting application dumps showed:
  • Most threads trying to execute a query are waiting in a lock in DbConnectionPool.GetObject
  • One thread is waiting in a lock in DbConnectionPoolGroup.a (see full callstack below)
  • Another thread is waiting in a lock in DbConnectionPool.a (see other full callstack below)
Overall not a single connection is retrieved and all queries are blocked.

We have the issue occurring in two different versions of our application. One is using dotConnect for Oracle 8.4.333, the other uses version 8.5.506.6.

This looks a lot like a concurrency issue in connection management to me.

Best Regards,
Ronald

Callstack DbConnectionPoolGroup.a:

Code: Select all

ntdll.dll!NtWaitForMultipleObjects()	 
KERNELBASE.dll!WaitForMultipleObjectsEx()	 
[Managed to Native Transition]	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool A_0)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPoolGroup.a(object A_0)	 
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	 
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	 
mscorlib.dll!System.Threading.TimerQueueTimer.CallCallback()	 
mscorlib.dll!System.Threading.TimerQueueTimer.Fire()	 
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()	 
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()
Callstack DbConnectionPool.a

Code: Select all

ntdll.dll!NtWaitForMultipleObjects()	 
KERNELBASE.dll!WaitForMultipleObjectsEx()	 
[Managed to Native Transition]	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPool.a(Devart.Common.DbConnectionBase A_0)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPool.GetObject(Devart.Common.DbConnectionBase owningConnection)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionFactory.b(Devart.Common.DbConnectionBase A_0)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionClosed.Open(Devart.Common.DbConnectionBase outerConnection)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionBase.Open()	 
Devart.Data.Oracle.dll!Devart.Data.Oracle.OracleConnection.Open()	 
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.InternalDispatcher<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor>.Dispatch<System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext>(System.Data.Common.DbConnection target, System.Action<System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> operation, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext interceptionContext, System.Action<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor, System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> executing, System.Action<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor, System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> executed)	 
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(System.Data.Common.DbConnection connection, System.Data.Entity.Infrastructure.Interception.DbInterceptionContext interceptionContext)	 
EntityFramework.dll!System.Data.Entity.Core.EntityClient.EntityConnection.Open()	 
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.EnsureConnection(bool shouldMonitorTransactions)	 
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction<System.__Canon>(System.Func<System.__Canon> func, System.Data.Entity.Infrastructure.IDbExecutionStrategy executionStrategy, bool startLocalTransaction, bool releaseConnectionOnSuccess)

Re: Deadlock in ConnectionPool

Posted: Fri 11 Mar 2016 15:16
by Pinturiccio
Probably, very many connection with the same connection string is created under heavy load. Connection pool has a limit on the max number of connections in it, which can be changed via the MaxPoolSize connection property, the default value is 100. If a pool contains maximal allowed number of opened connections, it's not possible to add a connection to this pool. When creating such a connection, it will wait till one of the pool connections will be closed.

You can increase the max number of connections in a pool by assigning the corresponding value to the MaxPoolSize connection property or to the 'Max Pool Size' connection string parameter. For more information, please refer to
https://www.devart.com/dotconnect/oracl ... tring.html
https://www.devart.com/dotconnect/oracl ... lSize.html

Re: Deadlock in ConnectionPool

Posted: Fri 11 Mar 2016 17:01
by BGrojer
I missed to highlight one important aspect in the initial post:
There are no queries really being executed at the time of blockage.

All threads are waiting for ConnectionPool to provide a connection. There is not a single one actually owning and using a connection at that moment.

I am aware about MaxPoolSize, we intentionally use the default value 100. But the system behaves like MaxPoolSize was 0. No queries being executed, all threads/tasks wait in a lock inside ConnectionPool.

Re: Deadlock in ConnectionPool

Posted: Mon 14 Mar 2016 11:35
by Pinturiccio
Please increase MaxPoolSize to 1000 for all connections. Is the issue still reproduced under heavy load after this?

Please also set "Oci Session Pooling=true" in the connection string for all connections. Is the issue still reproduced with this change?

Re: Deadlock in ConnectionPool

Posted: Thu 17 Mar 2016 10:14
by BGrojer
Pinturiccio wrote:increase MaxPoolSize to 1000 for all connections
Changing MaxPoolSize to 1000 is more then our Oracle Server at the customer can handle (250 limit) and we have two applications running against oracle.
Pinturiccio wrote:Please also set "Oci Session Pooling=true" in the connection string for all connections.
We use DirectMode in DevArt. OCI is not used.

The only workaround that looks working currently is to modifiy MinPoolSize and MaxPoolSize to the same value. (Looks like the methods in DevArt for increasing, decreasing the pool are then not executed anymore). At least two customer haven't report the issue anymore since applying this change few days ago.

Please have a look on the provided call stack. There is a clear possiblity to run into a dead lock in DevArt code (DbConnectionPool and DbConnectionPoolGroup). We have process dumps from customers available that are showing that issues with ~ 50 threads stuck in those methods.

Re: Deadlock in ConnectionPool

Posted: Thu 17 Mar 2016 15:46
by Pinturiccio
We will check our pooling code for potential locks. When we will get any results we will notify you.

Re: Deadlock in ConnectionPool

Posted: Mon 04 Apr 2016 11:51
by BGrojer
Any News on that one?

Re: Deadlock in ConnectionPool

Posted: Thu 07 Apr 2016 11:22
by Pinturiccio
We are investigating the issue and will post here about the results in the nearest future.

Re: Deadlock in ConnectionPool

Posted: Tue 26 Apr 2016 14:25
by BGrojer
Any News on that one? I have seen that you guys touched ConnectionPool in 9.0.7. But this version has another major bug during inserts.

I assume that you are not planning to do further changes to 8.5.x ??

Re: Deadlock in ConnectionPool

Posted: Wed 27 Apr 2016 07:55
by Pinturiccio
BGrojer wrote:I assume that you are not planning to do further changes to 8.5.x ??
We don't make patches. New functionality and bug fixes are included into new builds of dotConnect for Oracle.
BGrojer wrote:I have seen that you guys touched ConnectionPool in 9.0.7.
Could you confirm that the issue with deadlock in connection pooling has gone?
BGrojer wrote:But this version has another major bug during inserts.
Please describe the issue in more details. If it is an exception, send us its message and stack trace. Please also describe the steps for reproducing the issue.

Re: Deadlock in ConnectionPool

Posted: Wed 27 Apr 2016 09:42
by BGrojer
Please describe the issue in more details. If it is an exception, send us its message and stack trace. Please also describe the steps for reproducing the issue.
Please see:
http://forums.devart.com/viewtopic.php?f=1&t=33581
All times are UTC
Page 1 of 1
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/