Deadlock in ConnectionPool

Deadlock in ConnectionPool

Postby BGrojer » Thu 10 Mar 2016 16:40

Hi,
we have been encountering regular ConnectionPool deadlocks under heavy load.

Inspecting application dumps showed:
  • Most threads trying to execute a query are waiting in a lock in DbConnectionPool.GetObject
  • One thread is waiting in a lock in DbConnectionPoolGroup.a (see full callstack below)
  • Another thread is waiting in a lock in DbConnectionPool.a (see other full callstack below)
Overall not a single connection is retrieved and all queries are blocked.

We have the issue occurring in two different versions of our application. One is using dotConnect for Oracle 8.4.333, the other uses version 8.5.506.6.

This looks a lot like a concurrency issue in connection management to me.

Best Regards,
Ronald

Callstack DbConnectionPoolGroup.a:
Code: Select all
ntdll.dll!NtWaitForMultipleObjects()   
KERNELBASE.dll!WaitForMultipleObjectsEx()   
[Managed to Native Transition]   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool A_0)   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPoolGroup.a(object A_0)   
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   
mscorlib.dll!System.Threading.TimerQueueTimer.CallCallback()   
mscorlib.dll!System.Threading.TimerQueueTimer.Fire()   
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()   
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()


Callstack DbConnectionPool.a
Code: Select all
ntdll.dll!NtWaitForMultipleObjects()   
KERNELBASE.dll!WaitForMultipleObjectsEx()   
[Managed to Native Transition]   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPool.a(Devart.Common.DbConnectionBase A_0)   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPool.GetObject(Devart.Common.DbConnectionBase owningConnection)   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionFactory.b(Devart.Common.DbConnectionBase A_0)   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionClosed.Open(Devart.Common.DbConnectionBase outerConnection)   
Devart.Data.Oracle.dll!Devart.Common.DbConnectionBase.Open()   
Devart.Data.Oracle.dll!Devart.Data.Oracle.OracleConnection.Open()   
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.InternalDispatcher<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor>.Dispatch<System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext>(System.Data.Common.DbConnection target, System.Action<System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> operation, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext interceptionContext, System.Action<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor, System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> executing, System.Action<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor, System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> executed)   
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(System.Data.Common.DbConnection connection, System.Data.Entity.Infrastructure.Interception.DbInterceptionContext interceptionContext)   
EntityFramework.dll!System.Data.Entity.Core.EntityClient.EntityConnection.Open()   
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.EnsureConnection(bool shouldMonitorTransactions)   
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction<System.__Canon>(System.Func<System.__Canon> func, System.Data.Entity.Infrastructure.IDbExecutionStrategy executionStrategy, bool startLocalTransaction, bool releaseConnectionOnSuccess)   
BGrojer
 
Posts: 18
Joined: Thu 19 Sep 2013 12:35

Re: Deadlock in ConnectionPool

Postby Pinturiccio » Fri 11 Mar 2016 15:16

Probably, very many connection with the same connection string is created under heavy load. Connection pool has a limit on the max number of connections in it, which can be changed via the MaxPoolSize connection property, the default value is 100. If a pool contains maximal allowed number of opened connections, it's not possible to add a connection to this pool. When creating such a connection, it will wait till one of the pool connections will be closed.

You can increase the max number of connections in a pool by assigning the corresponding value to the MaxPoolSize connection property or to the 'Max Pool Size' connection string parameter. For more information, please refer to
https://www.devart.com/dotconnect/oracle/docs/?Devart.Data.Oracle~Devart.Data.Oracle.OracleConnection~ConnectionString.html
https://www.devart.com/dotconnect/oracle/docs/?Devart.Data.Oracle~Devart.Data.Oracle.OracleConnectionStringBuilder~MaxPoolSize.html
Pinturiccio
Devart Team
 
Posts: 2021
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in ConnectionPool

Postby BGrojer » Fri 11 Mar 2016 17:01

I missed to highlight one important aspect in the initial post:
There are no queries really being executed at the time of blockage.

All threads are waiting for ConnectionPool to provide a connection. There is not a single one actually owning and using a connection at that moment.

I am aware about MaxPoolSize, we intentionally use the default value 100. But the system behaves like MaxPoolSize was 0. No queries being executed, all threads/tasks wait in a lock inside ConnectionPool.
BGrojer
 
Posts: 18
Joined: Thu 19 Sep 2013 12:35

Re: Deadlock in ConnectionPool

Postby Pinturiccio » Mon 14 Mar 2016 11:35

Please increase MaxPoolSize to 1000 for all connections. Is the issue still reproduced under heavy load after this?

Please also set "Oci Session Pooling=true" in the connection string for all connections. Is the issue still reproduced with this change?
Pinturiccio
Devart Team
 
Posts: 2021
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in ConnectionPool

Postby BGrojer » Thu 17 Mar 2016 10:14

Pinturiccio wrote:increase MaxPoolSize to 1000 for all connections

Changing MaxPoolSize to 1000 is more then our Oracle Server at the customer can handle (250 limit) and we have two applications running against oracle.

Pinturiccio wrote:Please also set "Oci Session Pooling=true" in the connection string for all connections.

We use DirectMode in DevArt. OCI is not used.

The only workaround that looks working currently is to modifiy MinPoolSize and MaxPoolSize to the same value. (Looks like the methods in DevArt for increasing, decreasing the pool are then not executed anymore). At least two customer haven't report the issue anymore since applying this change few days ago.

Please have a look on the provided call stack. There is a clear possiblity to run into a dead lock in DevArt code (DbConnectionPool and DbConnectionPoolGroup). We have process dumps from customers available that are showing that issues with ~ 50 threads stuck in those methods.
BGrojer
 
Posts: 18
Joined: Thu 19 Sep 2013 12:35

Re: Deadlock in ConnectionPool

Postby Pinturiccio » Thu 17 Mar 2016 15:46

We will check our pooling code for potential locks. When we will get any results we will notify you.
Pinturiccio
Devart Team
 
Posts: 2021
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in ConnectionPool

Postby BGrojer » Mon 04 Apr 2016 11:51

Any News on that one?
BGrojer
 
Posts: 18
Joined: Thu 19 Sep 2013 12:35

Re: Deadlock in ConnectionPool

Postby Pinturiccio » Thu 07 Apr 2016 11:22

We are investigating the issue and will post here about the results in the nearest future.
Pinturiccio
Devart Team
 
Posts: 2021
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in ConnectionPool

Postby BGrojer » Tue 26 Apr 2016 14:25

Any News on that one? I have seen that you guys touched ConnectionPool in 9.0.7. But this version has another major bug during inserts.

I assume that you are not planning to do further changes to 8.5.x ??
BGrojer
 
Posts: 18
Joined: Thu 19 Sep 2013 12:35

Re: Deadlock in ConnectionPool

Postby Pinturiccio » Wed 27 Apr 2016 07:55

BGrojer wrote:I assume that you are not planning to do further changes to 8.5.x ??

We don't make patches. New functionality and bug fixes are included into new builds of dotConnect for Oracle.

BGrojer wrote:I have seen that you guys touched ConnectionPool in 9.0.7.

Could you confirm that the issue with deadlock in connection pooling has gone?

BGrojer wrote:But this version has another major bug during inserts.

Please describe the issue in more details. If it is an exception, send us its message and stack trace. Please also describe the steps for reproducing the issue.
Pinturiccio
Devart Team
 
Posts: 2021
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in ConnectionPool

Postby BGrojer » Wed 27 Apr 2016 09:42

Please describe the issue in more details. If it is an exception, send us its message and stack trace. Please also describe the steps for reproducing the issue.


Please see:
http://forums.devart.com/viewtopic.php?f=1&t=33581
BGrojer
 
Posts: 18
Joined: Thu 19 Sep 2013 12:35


Return to dotConnect for Oracle