Deadlock in ConnectionPool

[BGrojer]

Hi,
we have been encountering regular ConnectionPool deadlocks under heavy load.

Inspecting application dumps showed:

• Most threads trying to execute a query are waiting in a lock in DbConnectionPool.GetObject
• One thread is waiting in a lock in DbConnectionPoolGroup.a (see full callstack below)
• Another thread is waiting in a lock in DbConnectionPool.a (see other full callstack below)

Overall not a single connection is retrieved and all queries are blocked.

We have the issue occurring in two different versions of our application. One is using dotConnect for Oracle 8.4.333, the other uses version 8.5.506.6.

This looks a lot like a concurrency issue in connection management to me.

Best Regards,
Ronald

Callstack DbConnectionPoolGroup.a:

Code: Select all

ntdll.dll!NtWaitForMultipleObjects()	 
KERNELBASE.dll!WaitForMultipleObjectsEx()	 
[Managed to Native Transition]	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool A_0)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPoolGroup.a(object A_0)	 
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	 
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	 
mscorlib.dll!System.Threading.TimerQueueTimer.CallCallback()	 
mscorlib.dll!System.Threading.TimerQueueTimer.Fire()	 
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()	 
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()

Callstack DbConnectionPool.a

Code: Select all

ntdll.dll!NtWaitForMultipleObjects()	 
KERNELBASE.dll!WaitForMultipleObjectsEx()	 
[Managed to Native Transition]	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPool.a(Devart.Common.DbConnectionBase A_0)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionPool.GetObject(Devart.Common.DbConnectionBase owningConnection)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionFactory.b(Devart.Common.DbConnectionBase A_0)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionClosed.Open(Devart.Common.DbConnectionBase outerConnection)	 
Devart.Data.Oracle.dll!Devart.Common.DbConnectionBase.Open()	 
Devart.Data.Oracle.dll!Devart.Data.Oracle.OracleConnection.Open()	 
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.InternalDispatcher<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor>.Dispatch<System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext>(System.Data.Common.DbConnection target, System.Action<System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> operation, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext interceptionContext, System.Action<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor, System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> executing, System.Action<System.Data.Entity.Infrastructure.Interception.IDbConnectionInterceptor, System.Data.Common.DbConnection, System.Data.Entity.Infrastructure.Interception.DbConnectionInterceptionContext> executed)	 
EntityFramework.dll!System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(System.Data.Common.DbConnection connection, System.Data.Entity.Infrastructure.Interception.DbInterceptionContext interceptionContext)	 
EntityFramework.dll!System.Data.Entity.Core.EntityClient.EntityConnection.Open()	 
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.EnsureConnection(bool shouldMonitorTransactions)	 
EntityFramework.dll!System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction<System.__Canon>(System.Func<System.__Canon> func, System.Data.Entity.Infrastructure.IDbExecutionStrategy executionStrategy, bool startLocalTransaction, bool releaseConnectionOnSuccess)	 

[Pinturiccio]

Probably, very many connection with the same connection string is created under heavy load. Connection pool has a limit on the max number of connections in it, which can be changed via the MaxPoolSize connection property, the default value is 100. If a pool contains maximal allowed number of opened connections, it's not possible to add a connection to this pool. When creating such a connection, it will wait till one of the pool connections will be closed.

You can increase the max number of connections in a pool by assigning the corresponding value to the MaxPoolSize connection property or to the 'Max Pool Size' connection string parameter. For more information, please refer to
https://www.devart.com/dotconnect/oracl ... tring.html
https://www.devart.com/dotconnect/oracl ... lSize.html

[BGrojer]

I missed to highlight one important aspect in the initial post:
There are no queries really being executed at the time of blockage.

All threads are waiting for ConnectionPool to provide a connection. There is not a single one actually owning and using a connection at that moment.

I am aware about MaxPoolSize, we intentionally use the default value 100. But the system behaves like MaxPoolSize was 0. No queries being executed, all threads/tasks wait in a lock inside ConnectionPool.

[Pinturiccio]

Please increase MaxPoolSize to 1000 for all connections. Is the issue still reproduced under heavy load after this?

Please also set "Oci Session Pooling=true" in the connection string for all connections. Is the issue still reproduced with this change?

[BGrojer]

increase MaxPoolSize to 1000 for all connections

Changing MaxPoolSize to 1000 is more then our Oracle Server at the customer can handle (250 limit) and we have two applications running against oracle.

Please also set "Oci Session Pooling=true" in the connection string for all connections.

We use DirectMode in DevArt. OCI is not used.

The only workaround that looks working currently is to modifiy MinPoolSize and MaxPoolSize to the same value. (Looks like the methods in DevArt for increasing, decreasing the pool are then not executed anymore). At least two customer haven't report the issue anymore since applying this change few days ago.

Please have a look on the provided call stack. There is a clear possiblity to run into a dead lock in DevArt code (DbConnectionPool and DbConnectionPoolGroup). We have process dumps from customers available that are showing that issues with ~ 50 threads stuck in those methods.

[Pinturiccio]

We will check our pooling code for potential locks. When we will get any results we will notify you.

[BGrojer]

Any News on that one?

[Pinturiccio]

We are investigating the issue and will post here about the results in the nearest future.

[BGrojer]

Any News on that one? I have seen that you guys touched ConnectionPool in 9.0.7. But this version has another major bug during inserts.

I assume that you are not planning to do further changes to 8.5.x ??

[Pinturiccio]

I assume that you are not planning to do further changes to 8.5.x ??

We don't make patches. New functionality and bug fixes are included into new builds of dotConnect for Oracle.

I have seen that you guys touched ConnectionPool in 9.0.7.

Could you confirm that the issue with deadlock in connection pooling has gone?

But this version has another major bug during inserts.

Please describe the issue in more details. If it is an exception, send us its message and stack trace. Please also describe the steps for reproducing the issue.

[BGrojer]

Please describe the issue in more details. If it is an exception, send us its message and stack trace. Please also describe the steps for reproducing the issue.

Please see:
http://forums.devart.com/viewtopic.php?f=1&t=33581