Deadlock in connection pool code

Discussion of open issues, suggestions and bugs regarding ADO.NET provider for MySQL
Post Reply
jleblanc
Posts: 14
Joined: Tue 19 Apr 2016 22:31

Deadlock in connection pool code

Post by jleblanc » Wed 12 Oct 2016 22:22

Devart.Data.MySql, Version: 8.5.644.0

We were having performance issues with our database; it was extremely busy doing a lot of processing while our application was still trying to insert new data. We use connection pooling, and what happened is our connection pool maxed out and our application completely stalled.

None of our database commands were going through to MySQL and most of our apps processing completely stopped.

I took a memory dump of our application and did some investigation, and it looks like a deadlock occurred inside dotConnect's connection pooling code.

Here is a dump of all of the threads running in our application, truncated for brevity.

Code: Select all

0:252> !threads
ThreadCount:      485
UnstartedThread:  0
BackgroundThread: 473
PendingThread:    0
DeadThread:       2
Hosted Runtime:   no
        Lock  
       ID OSID ThreadOBJ           State GC Mode     GC Alloc Context  Domain           Count Apt Exception
   0    1 105c 0000000000434140    2a020 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA 
   2    2 130c 000000000043efb0    2b220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA (Finalizer) 
   4    3  390 0000000019342710  1020220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     Ukn (Threadpool Worker) 
   5    7 1f0c 00000000193739b0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000062513b0
   6    8 2528 00000000193ac770  102a220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA (Threadpool Worker) 
  22  249 1918 00000000193dbcc0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061c6c88
  23  425 14dc 000000002afdee20  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006265d60
  24  403 11d8 000000001b549530  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000622faa8
  25   19 1ffc 000000001b578550  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000062263e0
  26  124  384 000000002bb72810  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006263dd0
  27  279 1144 000000002add5ca0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005e017f0
  28  137 1064 000000002ad20d00  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000762ee28
  29  281 1414 000000002ad243b0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000055064c0
  30  136 10ac 000000002bcff650  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006283da8
  31  243 1e04 00000000193dd430  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000076578f0
  32  275  e48 00000000193ddc00  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061ed820
  33  203 1448 00000000193db4f0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007035e78
  34  419 18cc 000000002c357e10  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000621cd98
  35  247  e04 000000002aeb6870  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006276518
  36  152 1e20 000000002adc1af0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000761e320
  37  208 1894 000000002adc49d0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA (Threadpool Worker) 
  38  451 1700 000000002bb93c60  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000763f278
  39  313 1f08 000000002bb91d20  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006506378
  40  270 1dc8 000000002bb8cf00  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000076474a0
  41  394  b20 000000002bc10c60  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061ebae0
  42  337 1418 000000002bc11c00  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000076167b0
  43  402 13d8 000000002b4d8c90  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000004c78370
  44   92  8b4 000000002af7d5f0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000532dc18
  45  274 10cc 000000002bcd6710  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005ba3438
  46  212 2354 000000002ae74ad0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007637050
  47  424 23b4 000000002b025650  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005ea01c8
  48  205 1768 000000002adba410  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061ddd88
  49  202 1790 000000001b63a260  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005356730
  50  284 26a8 000000002bcd6ee0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000625bba8
  51   11  df0 000000002ad58410  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000760e588
  52  154 1a00 000000001b558590  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005d7c2d8
  53  222 2050 000000001b55a4d0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007626548
  54  251 21c0 000000002b3f4f20  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005e89620
  55  259 1cd8 000000002b0bbe50  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005c5bbc0
  56   47 24c8 000000001b553dc0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000765fb18
  57  277 276c 000000001b5e72c0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA (Threadpool Worker) 
  58  401  f5c 000000002c39ca70  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000063ff180
 199  172 1a24 000000002b2c4cb0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007043428
 200  113 1798 000000002b2c3540  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070511a8
 201  311 19e4 000000002b2c1dd0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070593d0
 202  458 25b0 000000002b2c2d70  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007086168
 203   22 1714 000000002b2c25a0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000709cbe0
 204   75 227c 000000002b2c5480  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070ba380
 205  340  ecc 000000002b3f2810  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070deb20
 206  339 21d8 000000002b3f37b0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007102f40
 207  346 2668 000000002b3f56f0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000714b090
 246  343  25c 000000002ad48470  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 1     MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007606360
 247   54 27f8 000000002afdfdc0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA (Threadpool Worker) 
 248  356 25fc 000000001b6bb3a0  3029220 Preemptive  0000000000000000:0000000000000000 0000000000425490 0     MTA (Threadpool Worker) 
You can see that most of them have thrown an exception, which is

Code: Select all

0:252> !PrintException /d 00000000061c6c88
Exception object: 00000000061c6c88
Exception type:   Devart.Data.MySql.MySqlException
Message:          Lost connection to MySQL server during query
InnerException:   System.TimeoutException, Use !PrintException 00000000061c6ad8 to see more.
StackTrace (generated):
    SP               IP               Function
    0000000021646E10 000007FE99A5BB9C Devart_Data_MySql!Devart.Data.MySql.ao.d(System.Exception)+0x9c
    0000000021646E50 000007FE96D63242 Devart_Data_MySql!Devart.Common.t.d(Byte[], Int32, Int32)+0xd2
    000000002164D0A0 000007FE96D63137 Devart_Data_MySql!Devart.Data.MySql.bi.a(Byte[], Int32, Int32)+0x2a7
    000000002164D110 000007FE96D62D62 Devart_Data_MySql!Devart.Data.MySql.bi.p()+0x22
    000000002164D160 000007FE96D62C99 Devart_Data_MySql!Devart.Data.MySql.bi.i()+0x29
    000000002164D1B0 000007FE96F9B1F4 Devart_Data_MySql!Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)+0x2b4
    000000002164D220 000007FE96F9AEA9 Devart_Data_MySql!Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)+0xf9
    000000002164D290 000007FE96F97247 Devart_Data_MySql!Devart.Data.MySql.g.e()+0xf7
    000000002164D310 000007FE96F9620C Devart_Data_MySql!Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)+0x34c

StackTraceString: <none>
HResult: 80004005
Using the SOSEX (http://www.stevestechspot.com) extension for WinDbg I did a deadlock detection which yielded

Code: Select all

0:000> !dlk
Examining SyncBlocks...
Scanning for ReaderWriterLock(Slim) instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Scanning for threads waiting on SyncBlocks...
*** ERROR: Module load completed but symbols could not be loaded for Devart.Data.MySql.dll
Scanning for threads waiting on ReaderWriterLock locks...
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*** WARNING: Unable to verify checksum for System.ServiceModel.Internals.ni.dll
Scanning for threads waiting on ReaderWriterLocksSlim locks...
*** WARNING: Unable to verify checksum for System.Data.ni.dll
Scanning for threads waiting on CriticalSections...
*** WARNING: Unable to verify checksum for System.ni.dll
*** WARNING: Unable to verify checksum for System.ServiceProcess.ni.dll
WARNING: Thread 2970 exited during dump writing (open status NTSTATUS 0xC000000B)
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
*DEADLOCK DETECTED*
CLR thread 0x4c holds the lock on SyncBlock 000000002bceecd8 OBJ:000000000367ee20[System.Collections.ArrayList]
...and is waiting for the lock on SyncBlock 000000002bceec38 OBJ:0000000005e31a28[Devart.Data.MySql.MySqlInternalConnection]
CLR thread 0xcd holds the lock on SyncBlock 000000002bceec38 OBJ:0000000005e31a28[Devart.Data.MySql.MySqlInternalConnection]
...and is waiting for the lock on SyncBlock 000000002bceecd8 OBJ:000000000367ee20[System.Collections.ArrayList]
CLR Thread 0x4c is waiting at Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool)(+0x25 IL,+0xa2 Native)
CLR Thread 0xcd is waiting at Devart.Common.DbConnectionPool.RemoveObject(System.Object)(+0x8 IL,+0x31 Native)


1 deadlock detected.
Next, I listed all threads that are waiting for locks, and which lock they are waiting for (truncated for brevity).

Code: Select all

0:000> !mwaits
Examining SyncBlocks...
Scanning for ReaderWriterLock(Slim) instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Scanning for threads waiting on SyncBlocks...
Scanning for threads waiting on ReaderWriterLock locks...
Scanning for threads waiting on ReaderWriterLocksSlim locks...
Scanning for threads waiting on CriticalSections...
WARNING: Thread 2970 exited during dump writing (open status NTSTATUS 0xC000000B)
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
Scanning threads for waits on unknown locks...
WARNING: Thread 2970 exited during dump writing (open status NTSTATUS 0xC000000B)
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB

ClrThread  DbgThread  OsThread    LockType    Lock              LockLevel
------------------------------------------------------------------------------
0xf        75         0x15c       SyncBlock   0000000019325cc8
0x1a2      331        0x178       SyncBlock   0000000019325cc8
0xe6       117        0x1d0       SyncBlock   000000002bceecd8
0x8a       359        0x248       SyncBlock   0000000019325cc8
0x157      246        0x25c       SyncBlock   000000002bceecd8
0x7e       341        0x26c       SyncBlock   0000000019325cc8
0x168      154        0x2d0       SyncBlock   000000002bceecd8
0x10a      449        0x2e0       SyncBlock   0000000019325cc8
0x6d       126        0x2f0       SyncBlock   000000002bceecd8
0x5e       444        0x308       SyncBlock   0000000019325cc8
0xb2       234        0x34c       SyncBlock   000000002bceecd8
0x7c       26         0x384       SyncBlock   000000002bceecd8
0x3        4          0x390       <Unknown>   0000000000000000
0x11a      307        0x3a0       SyncBlock   0000000019325cc8
0x9e       257        0x3ac       SyncBlock   0000000019325cc8
0x6b       215        0x3b4       SyncBlock   000000002bceecd8
0x108      278        0x3d0       SyncBlock   0000000019325cc8
0xe9       161        0x478       SyncBlock   000000002bceecd8
0x196      279        0x47c       SyncBlock   0000000019325cc8
0x10d      472        0x484       SyncBlock   0000000019325cc8
0x135      262        0x490       SyncBlock   0000000019325cc8
0x1d4      171        0x510       SyncBlock   000000002bceecd8
0x1ac      181        0x534       SyncBlock   000000002bceecd8
0x6a       216        0x53c       SyncBlock   000000002bceecd8
0x16f      108        0x560       SyncBlock   000000002bceecd8
0x1b3      260        0x594       SyncBlock   0000000019325cc8
0xd8       402        0x5ac       SyncBlock   0000000019325cc8
0x1dd      478        0x5e0       SyncBlock   0000000019325cc8
0x102      421        0x5ec       SyncBlock   0000000019325cc8
0x1f       363        0x5f4       SyncBlock   0000000019325cc8
0x1a7      185        0x63c       SyncBlock   000000002bceecd8
0x51       132        0x640       SyncBlock   000000002bceecd8
0x54       70         0x654       SyncBlock   0000000019325cc8
0x1d3      470        0x674       SyncBlock   0000000019325cc8
0x0        10         0x680       <Unknown>   0000000000000000
0x17c      469        0x6e4       SyncBlock   0000000019325cc8
0x18c      107        0x6e8       SyncBlock   000000002bceecd8
0x81       114        0x6fc       SyncBlock   000000002bceecd8
0xe7       230        0x728       SyncBlock   000000002bceecd8
0x1d1      62         0x72c       SyncBlock   000000002bceecd8
0x1b6      217        0x744       SyncBlock   000000002bceecd8
0xbc       101        0x798       SyncBlock   000000002bceecd8
0xe5       94         0x7fc       SyncBlock   000000002bceecd8
0xdf       263        0x810       SyncBlock   0000000019325cc8
0x19c      142        0x81c       SyncBlock   000000002bceecd8
0x6e       452        0x820       SyncBlock   0000000019325cc8
0x55       447        0x824       SyncBlock   0000000019325cc8
0x172      329        0x828       SyncBlock   0000000019325cc8
0x8c       273        0xe18       SyncBlock   0000000019325cc8
0xb7       349        0xe28       SyncBlock   0000000019325cc8
0x11d      131        0xe2c       SyncBlock   000000002bceecd8
0x12b      68         0xe38       SyncBlock   000000002bceecd8
0xd2       112        0xe40       SyncBlock   000000002bceecd8
0x113      32         0xe48       SyncBlock   000000002bceecd8
0x11f      425        0xe88       SyncBlock   0000000019325cc8
0xb4       265        0xe9c       SyncBlock   0000000019325cc8
0x154      205        0xecc       SyncBlock   000000002bceecd8
0x21       21         0xed4       <Unknown>   0000000000000000
0x129      429        0xf14       SyncBlock   0000000019325cc8
0x4e       232        0xf24       SyncBlock   000000002bceecd8
0xf2       288        0xf40       SyncBlock   0000000019325cc8
0x68       92         0xf58       SyncBlock   000000002bceecd8
0x191      58         0xf5c       SyncBlock   000000002bceecd8
0x6c       474        0xfd8       SyncBlock   0000000019325cc8
0x2d       335        0x1004      SyncBlock   0000000019325cc8
0x15d      140        0x100c      SyncBlock   000000002bceecd8
0x123      391        0x1010      SyncBlock   0000000019325cc8
0xa        98         0x1014      SyncBlock   000000002bceecd8
0x173      144        0x103c      SyncBlock   000000002bceecd8
0x199      295        0x1050      SyncBlock   0000000019325cc8
0x1        0          0x105c      <Unknown>   0000000000000000
0x89       28         0x1064      SyncBlock   000000002bceecd8
0xb0       371        0x1088      SyncBlock   0000000019325cc8
0xbb       440        0x1094      SyncBlock   0000000019325cc8
0x88       30         0x10ac      SyncBlock   000000002bceecd8
0x19b      398        0x10b8      SyncBlock   0000000019325cc8
0x187      303        0x10c0      SyncBlock   0000000019325cc8
0x112      45         0x10cc      SyncBlock   000000002bceecd8
0x0        9          0x10e8      <Unknown>   0000000000000000
0x56       208        0x1118      SyncBlock   000000002bceecd8
0x1c9      244        0x1130      SyncBlock   000000002bceecd8
0x117      27         0x1144      SyncBlock   000000002bceecd8
0x15e      227        0x1150      SyncBlock   000000002bceecd8
0x32       280        0x116c      SyncBlock   0000000019325cc8
0x1cb      290        0x1174      SyncBlock   0000000019325cc8
0x1a5      351        0x117c      SyncBlock   0000000019325cc8
0x109      175        0x1184      SyncBlock   000000002bceecd8
0x50       450        0x1188      SyncBlock   0000000019325cc8
0x16e      266        0x1198      SyncBlock   0000000019325cc8
0x142      308        0x11a8      SyncBlock   0000000019325cc8
0x1a4      59         0x11b0      SyncBlock   000000002bceecd8
0xa1       313        0x11b4      SyncBlock   0000000019325cc8
0x18b      343        0x11b8      SyncBlock   0000000019325cc8
0x193      24         0x11d8      SyncBlock   000000002bceecd8
0xc5       113        0x11e0      SyncBlock   000000002bceecd8
0xa6       323        0x11f4      SyncBlock   0000000019325cc8
0x1d0      314        0x11f8      SyncBlock   0000000019325cc8
0xee       102        0x1224      SyncBlock   000000002bceecd8
0x7d       166        0x123c      SyncBlock   000000002bceecd8
0x16a      231        0x124c      SyncBlock   000000002bceecd8
0xcf       264        0x1268      SyncBlock   0000000019325cc8
0x120      173        0x1280      SyncBlock   000000002bceecd8
0xd3       416        0x129c      SyncBlock   0000000019325cc8
0x2a       370        0x12b4      SyncBlock   0000000019325cc8
0x16b      404        0x12bc      SyncBlock   0000000019325cc8
0x5a       225        0x12f4      SyncBlock   000000002bceecd8
0x11b      379        0x1300      SyncBlock   0000000019325cc8
0x2        2          0x130c      <Unknown>   0000000000000000
0x30       268        0x1334      SyncBlock   0000000019325cc8
0xf6       196        0x1348      SyncBlock   000000002bceecd8
0x1e3      481        0x1350      <Unknown>   0000000000000000
0x38       88         0x135c      SyncBlock   000000002bceecd8
0xed       414        0x136c      SyncBlock   0000000019325cc8
0x52       415        0x1370      SyncBlock   0000000019325cc8
0x114      282        0x1378      SyncBlock   0000000019325cc8
0x1ba      220        0x13a4      SyncBlock   000000002bceecd8
0x18e      150        0x13bc      SyncBlock   000000002bceecd8
0x4c       252        0x13d4      SyncBlock   000000002bceec38
0x192      43         0x13d8      SyncBlock   000000002bceecd8
0x84       147        0x13dc      SyncBlock   000000002bceecd8
0x8e       348        0x13e4      SyncBlock   0000000019325cc8
0x4a       468        0x13f0      SyncBlock   0000000019325cc8
0x24       76         0x140c      SyncBlock   000000002bceecd8
0x119      29         0x1414      SyncBlock   000000002bceecd8
0x151      42         0x1418      SyncBlock   000000002bceecd8
0xcb       33         0x1448      SyncBlock   000000002bceecd8
0x4f       448        0x1450      SyncBlock   0000000019325cc8
0x5d       382        0x1460      SyncBlock   0000000019325cc8
0x194      96         0x146c      SyncBlock   000000002bceecd8
0xb5       397        0x147c      SyncBlock   0000000019325cc8
0x1ce      410        0x16e0      SyncBlock   0000000019325cc8
0x91       187        0x16e4      SyncBlock   000000002bceecd8
0x1c3      38         0x1700      SyncBlock   000000002bceecd8
0x16       203        0x1714      SyncBlock   000000002bceecd8
0x1b4      345        0x1750      SyncBlock   0000000019325cc8
0x13a      328        0x1764      SyncBlock   0000000019325cc8
0xcd       48         0x1768      SyncBlock   000000002bceecd8
0x13d      153        0x1770      SyncBlock   000000002bceecd8
0x26       350        0x1774      SyncBlock   0000000019325cc8
0x8d       103        0x1788      SyncBlock   000000002bceecd8
0xca       49         0x1790      SyncBlock   000000002bceecd8
0x71       200        0x1798      SyncBlock   000000002bceecd8
0xc8       310        0x179c      SyncBlock   0000000019325cc8
0x10       184        0x17ac      SyncBlock   000000002bceecd8
0x78       236        0x17c0      SyncBlock   000000002bceecd8
0x15c      374        0x17e8      SyncBlock   0000000019325cc8
0x195      432        0x17ec      SyncBlock   0000000019325cc8
0x16d      419        0x1804      SyncBlock   0000000019325cc8
0x17       368        0x1814      SyncBlock   0000000019325cc8
0x1e6      479        0x1818      SyncBlock   0000000019325cc8
0xc7       136        0x1830      SyncBlock   000000002bceecd8
0x99       296        0x1838      SyncBlock   0000000019325cc8
0x19a      455        0x183c      SyncBlock   0000000019325cc8
0x150      428        0x1840      SyncBlock   0000000019325cc8
0xa0       165        0x1844      SyncBlock   000000002bceecd8
0xbd       275        0x1848      SyncBlock   0000000019325cc8
0xfe       457        0x184c      SyncBlock   0000000019325cc8
0x179      254        0x1850      SyncBlock   0000000019325cc8
0xab       462        0x1854      SyncBlock   0000000019325cc8
0xda       436        0x185c      SyncBlock   0000000019325cc8
0xef       369        0x1874      SyncBlock   0000000019325cc8
0x44       241        0x188c      SyncBlock   000000002bceecd8
0x133      211        0x1890      SyncBlock   000000002bceecd8
0xd0       37         0x1894      SyncBlock   0000000019325cc8
0xdc       145        0x1898      SyncBlock   000000002bceecd8
0x15f      80         0x18a0      SyncBlock   000000002bceecd8
0x5f       83         0x18ac      SyncBlock   000000002bceecd8
0xd5       218        0x18bc      SyncBlock   000000002bceecd8
0x1c4      77         0x18c0      SyncBlock   000000002bceecd8
0x1a3      34         0x18cc      SyncBlock   000000002bceecd8
0x149      411        0x1908      SyncBlock   0000000019325cc8
0xf9       22         0x1918      SyncBlock   000000002bceecd8
0xd7       441        0x1948      SyncBlock   0000000019325cc8
0x111      180        0x1950      SyncBlock   000000002bceecd8
0x12c      309        0x1954      SyncBlock   0000000019325cc8
0x20       20         0x1964      SyncBlock   0000000019325cc8
0x1cf      430        0x19a4      SyncBlock   0000000019325cc8
0x7f       212        0x19a8      SyncBlock   000000002bceecd8
0x66       129        0x19cc      SyncBlock   000000002bceecd8
0xa9       281        0x19d8      SyncBlock   0000000019325cc8
0x85       195        0x19dc      SyncBlock   000000002bceecd8
0x137      201        0x19e4      SyncBlock   000000002bceecd8
0x19       16         0x19e8      <Unknown>   0000000000000000
0x6f       365        0x19f4      SyncBlock   0000000019325cc8
0x4d       394        0x19f8      SyncBlock   0000000019325cc8
0x9a       52         0x1a00      SyncBlock   000000002bceecd8
0x147      120        0x1a04      SyncBlock   000000002bceecd8
0x79       321        0x1a10      SyncBlock   0000000019325cc8
0xac       199        0x1a24      SyncBlock   000000002bceecd8
0x80       332        0x1a40      SyncBlock   0000000019325cc8
0x11e      253        0x1a50      SyncBlock   0000000019325cc8
0x5b       276        0x1a74      SyncBlock   0000000019325cc8
0x1c0      118        0x1a80      SyncBlock   000000002bceecd8
0x144      137        0x1a8c      SyncBlock   000000002bceecd8
0x105      209        0x1ac0      SyncBlock   000000002bceecd8
0x69       157        0x1ac4      SyncBlock   000000002bceecd8
0x148      190        0x1ac8      SyncBlock   000000002bceecd8
0x6        116        0x1ad8      SyncBlock   000000002bceecd8
0x0        3          0x1af0      <Unknown>   0000000000000000
0x77       380        0x1af4      SyncBlock   0000000019325cc8
0x1a1      163        0x1b00      SyncBlock   000000002bceecd8
0x132      261        0x1b08      SyncBlock   0000000019325cc8
0x12f      123        0x1b24      SyncBlock   000000002bceecd8
0xc0       467        0x1b4c      SyncBlock   0000000019325cc8
0x1c6      176        0x1d3c      SyncBlock   000000002bceecd8
0xba       97         0x1d40      SyncBlock   000000002bceecd8
0x2b       121        0x1d44      SyncBlock   000000002bceecd8
0x1a0      298        0x1da0      SyncBlock   0000000019325cc8
0xc6       458        0x1db4      SyncBlock   0000000019325cc8
0x31       192        0x1db8      SyncBlock   000000002bceecd8
0xaf       460        0x1dc0      SyncBlock   0000000019325cc8
0x37       259        0x1dc4      SyncBlock   0000000019325cc8
0x10e      40         0x1dc8      SyncBlock   000000002bceecd8
0x14b      339        0x1df8      SyncBlock   0000000019325cc8
0xf3       31         0x1e04      SyncBlock   000000002bceecd8
0x188      93         0x1e08      SyncBlock   000000002bceecd8
0x98       36         0x1e20      SyncBlock   000000002bceecd8
0x74       322        0x1e38      SyncBlock   0000000019325cc8
0x1d       352        0x1e3c      SyncBlock   0000000019325cc8
0xb6       423        0x1e44      SyncBlock   0000000019325cc8
0xa7       130        0x1e5c      SyncBlock   000000002bceecd8
0x5        162        0x1e60      SyncBlock   000000002bceecd8
0x189      270        0x1e6c      SyncBlock   0000000019325cc8
0x0        8          0x1e70      <Unknown>   0000000000000000
0x76       406        0x1e80      SyncBlock   0000000019325cc8
0x33       182        0x1e84      SyncBlock   000000002bceecd8
0x75       213        0x1e94      SyncBlock   000000002bceecd8
0x9d       69         0x1eb4      SyncBlock   000000002bceecd8
0x143      318        0x1ef8      SyncBlock   0000000019325cc8
0x139      39         0x1f08      SyncBlock   000000002bceecd8
0x7        5          0x1f0c      SyncBlock   000000002bceecd8
0x1b5      119        0x1f10      SyncBlock   000000002bceecd8
0x1bc      143        0x1f24      SyncBlock   000000002bceecd8
0x1e8      489        0x1f34      <Unknown>   0000000000000000
0x3c       71         0x1f50      SyncBlock   000000002bceecd8
0x96       302        0x1f5c      SyncBlock   0000000019325cc8
0x1d9      471        0x1f64      SyncBlock   0000000019325cc8
0xae       274        0x1f78      SyncBlock   0000000019325cc8
0x104      473        0x1f8c      SyncBlock   0000000019325cc8
0x156      446        0x1fa4      SyncBlock   0000000019325cc8
0xce       272        0x1fc0      SyncBlock   0000000019325cc8
0x1c2      194        0x1fd0      SyncBlock   000000002bceecd8
0x39       442        0x1fe4      SyncBlock   0000000019325cc8
0x1ad      300        0x1ff4      SyncBlock   0000000019325cc8
0x13       25         0x1ffc      SyncBlock   000000002bceecd8
0x0        12         0x2004      <Unknown>   0000000000000000
0xf5       445        0x202c      SyncBlock   0000000019325cc8
0xde       53         0x2050      SyncBlock   000000002bceecd8
0x4        357        0x2068      SyncBlock   0000000019325cc8
0x3d       168        0x206c      SyncBlock   000000002bceecd8
0x136      453        0x2070      SyncBlock   0000000019325cc8
0x159      367        0x2074      SyncBlock   0000000019325cc8
0x59       66         0x2080      SyncBlock   000000002bceecd8
0x9        111        0x2094      SyncBlock   000000002bceecd8
0x27       360        0x20a0      SyncBlock   0000000019325cc8
0xe1       172        0x20a4      SyncBlock   000000002bceecd8
0x130      221        0x20bc      SyncBlock   000000002bceecd8
0x47       90         0x20dc      SyncBlock   000000002bceecd8
0x1c1      222        0x20f4      SyncBlock   000000002bceecd8
0x14c      85         0x210c      SyncBlock   000000002bceecd8
0x10f      167        0x2118      SyncBlock   000000002bceecd8
0x1be      294        0x211c      SyncBlock   0000000019325cc8
0xa2       267        0x213c      SyncBlock   0000000019325cc8
0x122      81         0x2150      SyncBlock   000000002bceecd8
0x43       99         0x215c      SyncBlock   000000002bceecd8
0xd9       378        0x217c      SyncBlock   0000000019325cc8
0x3a       324        0x2194      SyncBlock   0000000019325cc8
0xe        297        0x2198      SyncBlock   0000000019325cc8
0x10b      346        0x21a4      SyncBlock   0000000019325cc8
0xd6       284        0x21b0      SyncBlock   0000000019325cc8
0xe2       233        0x21b8      SyncBlock   000000002bceecd8
0xfb       54         0x21c0      SyncBlock   000000002bceecd8
0x13f      189        0x21c8      SyncBlock   000000002bceecd8
0x153      206        0x21d8      SyncBlock   000000002bceecd8
0x13c      256        0x2210      SyncBlock   0000000019325cc8
0x126      386        0x2230      SyncBlock   0000000019325cc8
0x29       84         0x2240      SyncBlock   000000002bceecd8
0x145      239        0x2250      SyncBlock   000000002bceecd8
0xb1       424        0x2260      SyncBlock   0000000019325cc8
0x4b       204        0x227c      SyncBlock   000000002bceecd8
0x118      91         0x228c      SyncBlock   000000002bceecd8
0x183      475        0x2298      SyncBlock   0000000019325cc8
0x8f       353        0x22c8      SyncBlock   0000000019325cc8
0x162      243        0x22e4      SyncBlock   000000002bceecd8
0xc        340        0x22f4      SyncBlock   0000000019325cc8
0xd1       431        0x2328      SyncBlock   0000000019325cc8
0x163      106        0x232c      SyncBlock   000000002bceecd8
0x1c8      355        0x2338      SyncBlock   0000000019325cc8
0x34       229        0x2344      SyncBlock   000000002bceecd8
0x83       179        0x2348      SyncBlock   000000002bceecd8
0xd4       46         0x2354      SyncBlock   000000002bceecd8
0x1d7      361        0x2368      SyncBlock   0000000019325cc8
0x9b       396        0x237c      SyncBlock   0000000019325cc8
0x14       13         0x2394      SyncBlock   000000002bceecd8
0x165      128        0x2398      SyncBlock   000000002bceecd8
0x1a8      47         0x23b4      SyncBlock   000000002bceecd8
0x15b      223        0x23b8      SyncBlock   000000002bceecd8
0x1d8      433        0x23c0      SyncBlock   0000000019325cc8
0x46       198        0x23d4      SyncBlock   000000002bceecd8
0x1da      476        0x23d8      SyncBlock   0000000019325cc8
0x17d      139        0x23e0      SyncBlock   000000002bceecd8
0x0        1          0x23e4      <Unknown>   0000000000000000
0x1c       465        0x23ec      SyncBlock   0000000019325cc8
0x167      305        0x23f0      SyncBlock   0000000019325cc8
0xcc       127        0x23fc      SyncBlock   000000002bceecd8
0x166      319        0x241c      SyncBlock   0000000019325cc8
0x48       160        0x2420      SyncBlock   000000002bceecd8
0x1cd      292        0x244c      SyncBlock   0000000019325cc8
0x16c      413        0x245c      SyncBlock   0000000019325cc8
0x1aa      122        0x2460      SyncBlock   000000002bceecd8
0x9f       375        0x2478      SyncBlock   0000000019325cc8
0x25       403        0x2494      SyncBlock   0000000019325cc8
0x18f      327        0x249c      SyncBlock   0000000019325cc8
0xc3       342        0x24a0      SyncBlock   0000000019325cc8
0xfa       73         0x24a4      SyncBlock   000000002bceecd8
0x42       306        0x24a8      SyncBlock   0000000019325cc8
0xbe       438        0x24ac      SyncBlock   0000000019325cc8
0x2f       56         0x24c8      SyncBlock   000000002bceecd8
0x1d2      277        0x24cc      SyncBlock   0000000019325cc8
0x40       286        0x24f0      SyncBlock   0000000019325cc8
0x72       427        0x2500      SyncBlock   0000000019325cc8
0x1a6      422        0x2508      SyncBlock   0000000019325cc8
0x15       14         0x2518      <Unknown>   0000000000000000
0x106      409        0x2524      SyncBlock   0000000019325cc8
0x28       347        0x2530      SyncBlock   0000000019325cc8
0x1e       19         0x2544      <Unknown>   0000000000000000
0x17e      317        0x254c      SyncBlock   0000000019325cc8
0xe3       183        0x2570      SyncBlock   000000002bceecd8
0x7b       395        0x2594      SyncBlock   0000000019325cc8
0x12d      437        0x259c      SyncBlock   0000000019325cc8
0x1ca      202        0x25b0      SyncBlock   000000002bceecd8
0xad       169        0x25c4      SyncBlock   000000002bceecd8
0x2e       376        0x25d8      SyncBlock   0000000019325cc8
0x171      283        0x25e8      SyncBlock   0000000019325cc8
0x121      82         0x25ec      SyncBlock   000000002bceecd8
0x164      248        0x25fc      SyncBlock   0000000019325cc8
0x58       237        0x2604      SyncBlock   000000002bceecd8
0xc9       224        0x2628      SyncBlock   000000002bceecd8
0x185      439        0x2630      SyncBlock   0000000019325cc8
0xa4       337        0x2638      SyncBlock   0000000019325cc8
0x15a      207        0x2668      SyncBlock   000000002bceecd8
0x161      156        0x266c      SyncBlock   000000002bceecd8
0x1ae      63         0x2670      SyncBlock   000000002bceecd8
0xc2       356        0x2688      SyncBlock   0000000019325cc8
0x11c      50         0x26a8      SyncBlock   000000002bceecd8
0x1b8      242        0x26b0      SyncBlock   000000002bceecd8
Most threads are waiting for the 000000002bceecd8 SyncBlock, which is held by one of the deadlocked threads. I inspected the ArrayList that the deadlocked thread 0x4c is holding, and it contains MySqlInternalConnection objects.

As far as I can tell, none of our code is involved; I can provide the memory dump of our application as well as more detailed WinDbg output of my analysis, but those files are too large to attach here.
Last edited by jleblanc on Wed 12 Oct 2016 23:25, edited 1 time in total.

jleblanc
Posts: 14
Joined: Tue 19 Apr 2016 22:31

Re: Deadlock in connection pool code

Post by jleblanc » Wed 12 Oct 2016 23:23

Here is the callstack of one of the deadlocked threads

Code: Select all

0:252> !dumpstack
OS Thread Id: 0x13d4 (252)
Current frame: ntdll!NtWaitForMultipleObjects+0xa
Child-SP         RetAddr          Caller, Callee
00000000313dde60 000007fefd541430 KERNELBASE!WaitForMultipleObjectsEx+0xe8, calling ntdll!NtWaitForMultipleObjects
00000000313dde80 000007fef50a3719 (MethodDesc 000007fef4cf4780 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cf4790 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000313dde90 000007fef50ab254 (MethodDesc 000007fef4ceb9a8 +0x14 System.Environment.get_IsWinRTSupported()), calling clr!JIT_GetSharedNonGCStaticBaseNoCtor_InlineGetAppDomain
00000000313ddeb0 000007fef511890e (MethodDesc 000007fef4cf85d8 +0xae System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()), calling (MethodDesc 000007fef4cf85d8 +0xd1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run())
00000000313ddef0 000007fefd5419cc KERNELBASE!WaitForMultipleObjectsEx+0x224, calling ntdll!RtlActivateActivationContextUnsafeFast
00000000313ddf00 000007fef511ddce (MethodDesc 000007fef4dda9f0 +0x6e System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
00000000313ddf20 000007fef634a939 clr!JIT_GetSharedGCThreadStaticBase+0x99, calling clr!_security_check_cookie
00000000313ddf30 000007fef597fec4 (MethodDesc 000007fef4e2a1e8 +0x14 System.Threading.Tasks.Task.LogFinishCompletionNotification()), calling clr!JIT_GetSharedNonGCStaticBase_InlineGetAppDomain
00000000313ddf60 00000000773816d3 kernel32!WaitForMultipleObjectsExImplementation+0xb3, calling kernel32!WaitForMultipleObjectsEx
00000000313ddff0 000007fef6468976 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62, calling kernel32!WaitForMultipleObjectsExImplementation
00000000313de020 000007fef63d54b2 clr!Thread::GetFinalApartment+0x1a, calling clr!Thread::GetApartment
00000000313de040 000007fefd5410ac KERNELBASE!WaitForSingleObjectEx+0x79, calling ntdll!ZwWaitForSingleObject
00000000313de050 000007fef646877a clr!Thread::DoAppropriateWaitWorker+0x1d0, calling clr!WaitForMultipleObjectsEx_SO_TOLERANT
00000000313de150 000007fef6468591 clr!Thread::DoAppropriateWait+0x7d, calling clr!Thread::DoAppropriateWaitWorker
00000000313de1d0 000007fef646883d clr!CLREventBase::WaitEx+0xc0, calling clr!Thread::DoAppropriateWait
00000000313de210 000007fef648410d clr!AwareLock::AllocLockSemEvent+0x4e, calling kernel32!CreateEventWStub
00000000313de230 000007fef6343ee2 clr!FrameWithCookie<GCFrame>::FrameWithCookie<GCFrame>+0x42, calling clr!GetThread
00000000313de260 000007fef6483ed6 clr!AwareLock::EnterEpilogHelper+0xc6, calling clr!CLREventBase::Wait
00000000313de320 000007fef6483dea clr!AwareLock::EnterEpilog+0x62, calling clr!AwareLock::EnterEpilogHelper
00000000313de380 000007fef63d5002 clr!JITutil_MonEnterWorker+0xe2, calling clr!AwareLock::Enter
00000000313de460 000007fe983ef662 (MethodDesc 000007fe96f40188 +0xa2 Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool)), calling 000007fef63425b0 (stub for System.Threading.Monitor.Enter(System.Object))
00000000313de4f8 000007fef63d4f92 clr!JITutil_MonEnterWorker+0x72, calling clr!LazyMachStateCaptureState
00000000313de520 000007fe983ef662 (MethodDesc 000007fe96f40188 +0xa2 Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool)), calling 000007fef63425b0 (stub for System.Threading.Monitor.Enter(System.Object))
00000000313de550 000007fef50a2dba (MethodDesc 000007fef4c10350 +0x1a System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32)), calling 000007fef63462c0 (stub for System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32, Boolean))
00000000313de5e0 000007fe983ef282 (MethodDesc 000007fe96f40178 +0x102 Devart.Common.DbConnectionPoolGroup.a(System.Object)), calling (MethodDesc 000007fe96f40188 +0 Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool))
00000000313de680 000007fef50a39a5 (MethodDesc 000007fef4cf4790 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000313de7e0 000007fef50a3719 (MethodDesc 000007fef4cf4780 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cf4790 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000313de810 000007fef514be92 (MethodDesc 000007fef4ce2530 +0x172 System.Threading.TimerQueueTimer.CallCallback()), calling (MethodDesc 000007fef4cf4780 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000313de8a0 000007fef514bc7e (MethodDesc 000007fef4de39b0 +0x10e System.Threading.TimerQueueTimer.Fire()), calling (MethodDesc 000007fef4ce2530 +0 System.Threading.TimerQueueTimer.CallCallback())
00000000313de8b0 000007fef435d8dc (MethodDesc 000007fef3b72268 +0x9c System.Net.Dns.ResolveCallback(System.Object)), calling (MethodDesc 000007fef3c08e30 +0 System.Net.LazyAsyncResult.InvokeCallback(System.Object))
00000000313de910 000007fef50d2143 (MethodDesc 000007fef4ce2e50 +0x43 System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem())
00000000313de920 000007fef50d112b (MethodDesc 000007fef4de6538 +0x1b System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested()), calling clr!JIT_GetSharedNonGCStaticBase_InlineGetAppDomain
00000000313de960 000007fef50d136a (MethodDesc 000007fef4de6588 +0x1ea System.Threading.ThreadPoolWorkQueue.Dispatch()), calling 000007fe96ce6aa0
00000000313dea20 000007fef634a7f3 clr!CallDescrWorkerInternal+0x83
00000000313dea60 000007fef634a6de clr!CallDescrWorkerWithHandler+0x4a, calling clr!CallDescrWorkerInternal
00000000313deaa0 000007fef634ae76 clr!MethodDescCallSite::CallTargetWorker+0x251, calling clr!CallDescrWorkerWithHandler
00000000313deae0 000007fef634babb clr!REGUTIL::RegCacheValueNameSeenPerhaps+0x28, calling clr!HashiStringKnownLower80
00000000313deb10 000007fef634ba62 clr!REGUTIL::GetConfigInteger+0x62, calling clr!REGUTIL::RegCacheValueNameSeenPerhaps
00000000313deb30 000007fef634ad87 clr!MethodDescCallSite::CallTargetWorker+0xa1, calling clr!_chkstk
00000000313deb50 000007fef6349879 clr!ArgIteratorTemplate<ArgIteratorBase>::ForceSigWalk+0x16f, calling clr!ArgIteratorTemplate<ArgIteratorBase>::ComputeReturnFlags
00000000313deb90 000007fef6343605 clr!MetaSig::Init+0xc4, calling clr!SigParser::SkipExactlyOne
00000000313dec50 000007fef63d0221 clr!QueueUserWorkItemManagedCallback+0x2a, calling clr!MethodDescCallSite::CallTargetWorker
00000000313ded40 000007fef634c121 clr!ManagedThreadBase_DispatchInner+0x2d
00000000313ded80 000007fef634c0a8 clr!ManagedThreadBase_DispatchMiddle+0x6c, calling clr!ManagedThreadBase_DispatchInner
00000000313ded90 000007fefd55aa36 KERNELBASE!SetThreadStackGuarantee+0x46, calling KERNELBASE!GetSystemInfo
00000000313deda0 000007fefd543006 KERNELBASE!VirtualQuery+0x26, calling ntdll!ZwQueryVirtualMemory
00000000313dedc0 000007fef6265f08 MSVCR120_CLR0400!control87+0x28, calling MSVCR120_CLR0400!get_fpsr
00000000313dee20 000007fef63445f1 clr!Frame::Push+0xe, calling clr!GetThread
00000000313dee50 000007fef634af59 clr!FrameWithCookie<DebuggerU2MCatchHandlerFrame>::FrameWithCookie<DebuggerU2MCatchHandlerFrame>+0x26, calling clr!Frame::Push
00000000313dee80 000007fef634c019 clr!ManagedThreadBase_DispatchOuter+0x75, calling clr!ManagedThreadBase_DispatchMiddle
00000000313deea0 000007fef70214ef mscoreei!ShimLog::Log+0x1b8, calling mscoreei!_security_check_cookie
00000000313deec0 000007fef6490e60 clr!Thread::HasStarted+0x168, calling clr!CORDebuggerAttached
00000000313def10 000007fef634c15f clr!ManagedThreadBase_FullTransitionWithAD+0x2f, calling clr!ManagedThreadBase_DispatchOuter
00000000313def70 000007fef63d01ae clr!ManagedPerAppDomainTPCount::DispatchWorkItem+0xa2, calling clr!ManagedThreadBase_FullTransitionWithAD
00000000313def80 000007fef6407106 clr!SafeQueryInterface+0xd2
00000000313def90 000007fefddc84f3 ole32!CObjectContext::Release+0x13, calling ole32!CObjectContext::CommonRelease
00000000313defd0 000007fef64919aa clr!SetupThread+0x31b, calling clr!ClrFlsIncrementValue
00000000313defe0 000007fefdda8779 ole32!CObjectContext::GetCurrentThreadType+0x31, calling ole32!IsSTAThread
00000000313df030 000007fefddc24c5 ole32!CoInitializeEx+0x1ed, calling ole32!NotifyInitializeSpies
00000000313df070 000007fefddc0c82 ole32!CoUninitialize+0xbe, calling ole32!NotifyInitializeSpies
00000000313df0a0 000007fef648ff61 clr!Thread::SetApartment+0xf9, calling clr!Thread::GetApartment
00000000313df0c0 000007fef63cf0d3 clr!PerAppDomainTPCountList::GetAppDomainIndexForThreadpoolDispatch+0x6f
00000000313df100 000007fef63cf046 clr!ThreadpoolMgr::ExecuteWorkRequest+0x46
00000000313df130 000007fef63cef3a clr!ThreadpoolMgr::WorkerThreadStart+0xf4, calling clr!ThreadpoolMgr::ExecuteWorkRequest
00000000313df160 000007fef6343b2d clr!EEHeapFreeInProcessHeap+0x39, calling kernel32!HeapFree
00000000313df1c0 000007fef648fcb6 clr!Thread::intermediateThreadProc+0x7d
00000000313df200 000007fefd5432a0 KERNELBASE!GetModuleHandleForUnicodeString+0x20, calling ntdll!LdrGetDllHandle
00000000313df240 000007fefd543254 KERNELBASE!BasepGetModuleHandleExW+0x1f0, calling KERNELBASE!_security_check_cookie
00000000313df320 00000000775d25b5 ntdll!LdrGetProcedureAddressEx+0x40e, calling ntdll!_security_check_cookie
00000000313df490 00000000775aa698 ntdll!LdrpSnapThunk+0x325, calling ntdll!LdrGetProcedureAddressEx
00000000313df4d0 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
00000000313df570 00000000775d25b5 ntdll!LdrGetProcedureAddressEx+0x40e, calling ntdll!_security_check_cookie
00000000313df5e0 000007fefe221124 urlmon!TlsDllMain+0xf4, calling ntdll!RtlLeaveCriticalSection
00000000313df630 000007fefe7c1440 wininet!DllMain+0x57, calling wininet!AtlModule_DllMain
00000000313df680 000007fef7eb1132 wbemsvc!CRT_INIT+0x2af, calling wbemsvc!DllMain
00000000313df6e0 00000000775d2631 ntdll!LdrGetProcedureAddress+0x11, calling ntdll!LdrGetProcedureAddressEx
00000000313df710 000007fefd5434cb KERNELBASE!GetModuleHandleA+0x73, calling ntdll!RtlFreeUnicodeString
00000000313df750 000007fef19f7795 msvcr120!_calloc_impl+0x5d, calling ntdll!RtlAllocateHeap
00000000313df770 0000000077588631 ntdll!LdrpCallTlsInitializers+0x41, calling ntdll!RtlReleaseSRWLockShared
00000000313df7a0 0000000074b62d9d msvcr80!__CRTDLL_INIT+0x25d, calling kernel32!GetCurrentThreadIdStub
00000000313df7b0 000007fefc6e79a7 bcryptprimitives!DllMain+0x53, calling 000000000059004e
00000000313df7e0 00000000775aa3e2 ntdll!LdrpInitializeThread+0x22f, calling ntdll!RtlLeaveCriticalSection
00000000313df8a0 00000000775aa50a ntdll!LdrpInitializeThread+0x15e, calling ntdll!RtlActivateActivationContextUnsafeFast
00000000313df8a8 00000000775aa53b ntdll!LdrpInitializeThread+0x19a, calling ntdll!RtlDeactivateActivationContextUnsafeFast
00000000313df8e0 00000000775aa1f4 ntdll!LdrpInitialize+0xa4, calling ntdll!ZwTestAlert
00000000313df950 00000000775aa0f8 ntdll!LdrInitializeThunk+0x18, calling ntdll!NtContinue
00000000313dfdc0 000007fef648fc9f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
00000000313dfe00 00000000773759bd kernel32!BaseThreadInitThunk+0xd
00000000313dfe30 00000000775aa2e1 ntdll!RtlUserThreadStart+0x1d

jleblanc
Posts: 14
Joined: Tue 19 Apr 2016 22:31

Re: Deadlock in connection pool code

Post by jleblanc » Wed 12 Oct 2016 23:25

Here is the callstack of the other deadlocked thread, split over two posts because it is very long.

Code: Select all

0:048> !dumpstack
OS Thread Id: 0x1768 (48)
Current frame: ntdll!NtWaitForMultipleObjects+0xa
Child-SP         RetAddr          Caller, Callee
000000002f9d2510 000007fefd541430 KERNELBASE!WaitForMultipleObjectsEx+0xe8, calling ntdll!NtWaitForMultipleObjects
000000002f9d2540 000007feff821aa5 ws2_32!closesocket+0x1c5, calling ntdll!RtlFreeHeap
000000002f9d2560 000007fef634a832 clr!StubHelpers::SetLastError+0x13, calling clr!GetThread
000000002f9d25a0 000007fefd5419cc KERNELBASE!WaitForMultipleObjectsEx+0x224, calling ntdll!RtlActivateActivationContextUnsafeFast
000000002f9d2610 00000000773816d3 kernel32!WaitForMultipleObjectsExImplementation+0xb3, calling kernel32!WaitForMultipleObjectsEx
000000002f9d2660 000007fef3d9569c (MethodDesc 000007fef3b7edf0 +0x8c System.Net.SafeCloseSocket+InnerSafeCloseSocket.ReleaseHandle()), calling 000007fef3d2a4a0 (stub for System.Net.UnsafeNclNativeMethods+SafeNetHandles.closesocket(IntPtr))
000000002f9d2670 000007fef634a6de clr!CallDescrWorkerWithHandler+0x4a, calling clr!CallDescrWorkerInternal
000000002f9d26a0 000007fef6468976 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62, calling kernel32!WaitForMultipleObjectsExImplementation
000000002f9d26d0 000007fef63d54b2 clr!Thread::GetFinalApartment+0x1a, calling clr!Thread::GetApartment
000000002f9d26f0 000007fef634a6de clr!CallDescrWorkerWithHandler+0x4a, calling clr!CallDescrWorkerInternal
000000002f9d2700 000007fef646877a clr!Thread::DoAppropriateWaitWorker+0x1d0, calling clr!WaitForMultipleObjectsEx_SO_TOLERANT
000000002f9d2730 000007fef648e2ca clr!DispatchCallSimple+0x72, calling clr!_security_check_cookie
000000002f9d2800 000007fef6468591 clr!Thread::DoAppropriateWait+0x7d, calling clr!Thread::DoAppropriateWaitWorker
000000002f9d2880 000007fef646883d clr!CLREventBase::WaitEx+0xc0, calling clr!Thread::DoAppropriateWait
000000002f9d28d0 000007fef4a112f3 nlssorting!SortCompareString+0x2b3, calling nlssorting!_security_check_cookie
000000002f9d28e0 000007fef6343ee2 clr!FrameWithCookie<GCFrame>::FrameWithCookie<GCFrame>+0x42, calling clr!GetThread
000000002f9d2910 000007fef6483ed6 clr!AwareLock::EnterEpilogHelper+0xc6, calling clr!CLREventBase::Wait
000000002f9d2990 000007fef63d712a clr!Thread::HandleThreadAbort+0x66, calling clr!Thread::ReadyForAsyncException
000000002f9d29d0 000007fef6483dea clr!AwareLock::EnterEpilog+0x62, calling clr!AwareLock::EnterEpilogHelper
000000002f9d29e0 000007fef64841a2 clr!AwareLock::TryEnter+0x1e, calling clr!GetThread
000000002f9d2a30 000007fef6484521 clr!AwareLock::Contention+0x1e3, calling clr!AwareLock::Enter
000000002f9d2af0 000007fef6484293 clr!JITutil_MonContention+0xaf, calling clr!AwareLock::Contention
000000002f9d2bc0 000007fe99a60281 (MethodDesc 000007fe96f0dfd0 +0x31 Devart.Common.DbConnectionPool.RemoveObject(System.Object)), calling 000007fef63425b0 (stub for System.Threading.Monitor.Enter(System.Object))
000000002f9d2c58 000007fef6484245 clr!JITutil_MonContention+0x61, calling clr!LazyMachStateCaptureState
000000002f9d2c80 000007fe99a60281 (MethodDesc 000007fe96f0dfd0 +0x31 Devart.Common.DbConnectionPool.RemoveObject(System.Object)), calling 000007fef63425b0 (stub for System.Threading.Monitor.Enter(System.Object))
000000002f9d2cd0 000007fe96fafc5b (MethodDesc 000007fe96f0df20 +0x11b Devart.Common.DbConnectionPool.PutObject(System.Object)), calling (MethodDesc 000007fe96f0dfd0 +0 Devart.Common.DbConnectionPool.RemoveObject(System.Object))
000000002f9d2d40 000007fe96faf3cb (MethodDesc 000007fe96f0c388 +0x17b Devart.Common.DbConnectionInternal.CloseInternalConnection()), calling (MethodDesc 000007fe96f0df20 +0 Devart.Common.DbConnectionPool.PutObject(System.Object))
000000002f9d2d60 000007fe96d61e8a (MethodDesc 000007fe96f0e480 +0x4a Devart.Data.MySql.au.b(Devart.Data.MySql.ba)), calling 000007fef6342730 (stub for System.Threading.Monitor.Exit(System.Object))
000000002f9d2dd0 000007fe96faf1c4 (MethodDesc 000007fe96f0c390 +0x84 Devart.Common.DbConnectionInternal.Close())
000000002f9d2de0 000007fe96faf0df (MethodDesc 000007fe96f0eb88 +0xdf Devart.Data.MySql.MySqlInternalConnection.Close()), calling clr!JIT_GetFieldObj
000000002f9d2e20 000007fe96d52f47 (MethodDesc 000007fe96f0a2d8 +0x37 Devart.Common.DbConnectionBase.Close())
000000002f9d2e30 000007fe96d52e00 (MethodDesc 000007fe96f0ad90 +0x10 Devart.Data.MySql.MySqlConnection.l()), calling 000007fe96eb1670
000000002f9d2e60 000007fe99b61749 (MethodDesc 000007fe96f0ae30 +0xf9 Devart.Data.MySql.MySqlConnection.Kill(Devart.Data.MySql.MySqlConnection))
000000002f9d2e80 000007fe96d584d0 (MethodDesc 000007fe96f0ad70 +0x10 Devart.Data.MySql.MySqlConnection.Open()), calling 000007fe96eb1210
000000002f9d2eb0 000007fe99b610df (MethodDesc 000007fe96f42fb8 +0x13f Devart.Data.MySql.MySqlCommand.a(Boolean)), calling 000007fe98b97920
000000002f9d2ef0 000007fe99b60f6d (MethodDesc 000007fe96f48360 +0x2d Devart.Data.MySql.a8.a(System.Exception)), calling clr!JIT_IsInstanceOfClass
000000002f9d2f20 000007fe96f965dd (MethodDesc 000007fe96f42f08 +0x71d Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling 000007fe98b97840
000000002f9d2fa0 000007fef64959c5 clr!ExceptionTracker::CallHandler+0xc5
000000002f9d3040 000007fef64958c3 clr!ExceptionTracker::CallCatchHandler+0x7f, calling clr!ExceptionTracker::CallHandler
000000002f9d30a0 000007fef64933ed clr!GCCoopNoDtor::Enter+0xe, calling clr!GetThread
000000002f9d30d0 000007fef64942af clr!ProcessCLRException+0x2e6, calling clr!ExceptionTracker::CallCatchHandler
000000002f9d3118 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3180 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d31b0 00000000775a7f0d ntdll!RtlpExecuteHandlerForUnwind+0xd
000000002f9d31e0 0000000077597d4c ntdll!RtlUnwindEx+0x539, calling ntdll!RtlpExecuteHandlerForUnwind
000000002f9d3278 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3460 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3610 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d3688 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d36a0 000007fef634755f clr!RtlVirtualUnwind_Wrapper+0x6b, calling ntdll!RtlVirtualUnwind
000000002f9d36f0 000007fef63474e2 clr!Thread::VirtualUnwindNonLeafCallFrame+0x56, calling clr!RtlVirtualUnwind_Wrapper
000000002f9d3740 000007fef634734f clr!EECodeManager::GetGSCookieAddr+0x8f, calling clr!GcInfoDecoder::GcInfoDecoder
000000002f9d3780 000007fe96f94fd1 (MethodDesc 000007fe96f42788 +0x151 Devart.Common.DbCommandBase.ExecuteDbDataReader(System.Data.CommandBehavior, Boolean))
000000002f9d3880 000007fef6494348 clr!ClrUnwindEx+0x40, calling ntdll!RtlUnwindEx
000000002f9d39b0 000007fe96f94fd1 (MethodDesc 000007fe96f42788 +0x151 Devart.Common.DbCommandBase.ExecuteDbDataReader(System.Data.CommandBehavior, Boolean))
000000002f9d3b90 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9d3ca0 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d3cd0 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d3d00 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d3d10 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d3d20 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d3da0 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d3de8 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3e60 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3e80 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d3e88 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3eb0 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d3f48 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9d3f68 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d3f98 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d3fb0 000007fef634755f clr!RtlVirtualUnwind_Wrapper+0x6b, calling ntdll!RtlVirtualUnwind
000000002f9d4000 000007fef63474e2 clr!Thread::VirtualUnwindNonLeafCallFrame+0x56, calling clr!RtlVirtualUnwind_Wrapper
000000002f9d4050 000007fef634734f clr!EECodeManager::GetGSCookieAddr+0x8f, calling clr!GcInfoDecoder::GcInfoDecoder
000000002f9d4090 000007fe96f94fd1 (MethodDesc 000007fe96f42788 +0x151 Devart.Common.DbCommandBase.ExecuteDbDataReader(System.Data.CommandBehavior, Boolean))
000000002f9d4390 000007fe98768060 (MethodDesc 000007fe988d08e8 +0x7b0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fef4ed66b0 +0 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, mscorlib]].GetResult())
000000002f9d43f0 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d4560 00000000775988c8 ntdll!RtlRaiseException+0x48, calling ntdll!RtlCaptureContext
000000002f9d4590 0000000077598ac8 ntdll!RtlRaiseException+0x22f, calling ntdll!RtlDispatchException
000000002f9d46c0 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d4890 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d48c0 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d48e0 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d48f0 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d4900 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d4910 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d4a70 000007fe96f9aea9 (MethodDesc 000007fe96f474a8 +0xf9 Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)), calling (MethodDesc 000007fe96f47458 +0 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean))
000000002f9d4b30 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d4b50 000007fef648d85a clr!HndCreateHandle+0x112, calling clr!StressLog::LogOn
000000002f9d4b80 000007fef63fd315 clr!BaseDomain::CreateHandle+0x15, calling clr!GetCurrentThreadHomeHeapNumber
000000002f9d4bd0 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d4c08 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d4c50 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9d4c70 000007fef64952b4 clr!IsExceptionOfType+0x15, calling clr!Object::GetTrueMethodTable
000000002f9d4ca0 000007fef6495294 clr!RaiseTheExceptionInternalOnly+0x2fe, calling kernel32!RaiseExceptionStub
000000002f9d4da0 000007fef64a5347 clr!IL_Rethrow+0x98, calling clr!RaiseTheExceptionInternalOnly
000000002f9d4db0 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d4dc0 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d4dd0 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d4de0 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d4e40 000007fe96f9b1f4 (MethodDesc 000007fe96f47458 +0x2b4 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling clr!IL_Rethrow
000000002f9d4ed8 000007fef64a52ee clr!IL_Rethrow+0x3f, calling clr!LazyMachStateCaptureState
000000002f9d4ef0 000007fe96f9b1f4 (MethodDesc 000007fe96f47458 +0x2b4 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling clr!IL_Rethrow
000000002f9d4f60 000007fef64959c5 clr!ExceptionTracker::CallHandler+0xc5
000000002f9d5000 000007fef64958c3 clr!ExceptionTracker::CallCatchHandler+0x7f, calling clr!ExceptionTracker::CallHandler
000000002f9d5060 000007fef64933ed clr!GCCoopNoDtor::Enter+0xe, calling clr!GetThread
000000002f9d5090 000007fef64942af clr!ProcessCLRException+0x2e6, calling clr!ExceptionTracker::CallCatchHandler
000000002f9d50d8 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5140 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5170 00000000775a7f0d ntdll!RtlpExecuteHandlerForUnwind+0xd
000000002f9d51a0 0000000077597d4c ntdll!RtlUnwindEx+0x539, calling ntdll!RtlpExecuteHandlerForUnwind
000000002f9d5238 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5420 000007fe96f9aea9 (MethodDesc 000007fe96f474a8 +0xf9 Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)), calling (MethodDesc 000007fe96f47458 +0 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean))
000000002f9d5610 000007fe98768060 (MethodDesc 000007fe988d08e8 +0x7b0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fef4ed66b0 +0 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, mscorlib]].GetResult())
000000002f9d5630 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d5660 000007fef636edd7 clr!EEJitManager::JitCodeToMethodInfo+0x57, calling clr!EEJitManager::FindMethodCode
000000002f9d56b0 000007fef63d6fc4 clr!EECodeInfo::Init+0x6e
000000002f9d56c0 000007fef636ef53 clr!EECodeInfo::GetFunctionEntry+0x1d
000000002f9d56c8 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d5710 000007fef634755f clr!RtlVirtualUnwind_Wrapper+0x6b, calling ntdll!RtlVirtualUnwind
000000002f9d5740 000007fe96f9aea9 (MethodDesc 000007fe96f474a8 +0xf9 Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)), calling (MethodDesc 000007fe96f47458 +0 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean))
000000002f9d5840 000007fef6494348 clr!ClrUnwindEx+0x40, calling ntdll!RtlUnwindEx
000000002f9d5970 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5a30 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d5b50 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9d5c60 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d5c90 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d5cb0 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d5cc0 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d5cd0 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d5ce0 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d5d60 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d5da8 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5e20 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5e40 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d5e48 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5e70 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d5f08 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9d5f70 000007fef636edd7 clr!EEJitManager::JitCodeToMethodInfo+0x57, calling clr!EEJitManager::FindMethodCode
000000002f9d5fc0 000007fef63d6fc4 clr!EECodeInfo::Init+0x6e
000000002f9d5fd0 000007fef636ef53 clr!EECodeInfo::GetFunctionEntry+0x1d
000000002f9d5fd8 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d6020 000007fef634755f clr!RtlVirtualUnwind_Wrapper+0x6b, calling ntdll!RtlVirtualUnwind
000000002f9d6050 000007fe96f9aea9 (MethodDesc 000007fe96f474a8 +0xf9 Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)), calling (MethodDesc 000007fe96f47458 +0 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean))
000000002f9d6218 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d6230 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d6260 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d6290 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d62a0 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d6330 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d6350 000007fef64a57b1 clr!`ExceptionTracker::CallHandler'::`1'::filt$1+0x12, calling clr!SOTolerantBoundaryFilter
000000002f9d6378 00000000775a7f0d ntdll!RtlpExecuteHandlerForUnwind+0xd
000000002f9d6380 00000000775a7cce ntdll!RtlLookupFunctionTable+0xaa, calling ntdll!RtlReleaseSRWLockShared
000000002f9d63f0 000007fe98768060 (MethodDesc 000007fe988d08e8 +0x7b0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fef4ed66b0 +0 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, mscorlib]].GetResult())
000000002f9d6410 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d64f0 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d6520 00000000775988c8 ntdll!RtlRaiseException+0x48, calling ntdll!RtlCaptureContext
000000002f9d6550 0000000077598ac8 ntdll!RtlRaiseException+0x22f, calling ntdll!RtlDispatchException
000000002f9d6680 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d6900 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d6930 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d6950 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d6960 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d6970 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d6980 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d6ae0 000007fe96d63137 (MethodDesc 000007fe96f4aee0 +0x2a7 Devart.Data.MySql.bi.a(Byte[], Int32, Int32)), calling (MethodDesc 000007fe96f4bb58 +0 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d6b10 000007fef648d85a clr!HndCreateHandle+0x112, calling clr!StressLog::LogOn
000000002f9d6b40 000007fef63fd315 clr!BaseDomain::CreateHandle+0x15, calling clr!GetCurrentThreadHomeHeapNumber
000000002f9d6b90 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d6bc8 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d6c30 000007fef64952b4 clr!IsExceptionOfType+0x15, calling clr!Object::GetTrueMethodTable
000000002f9d6c60 000007fef6495294 clr!RaiseTheExceptionInternalOnly+0x2fe, calling kernel32!RaiseExceptionStub
000000002f9d6d60 000007fef649508e clr!IL_Throw+0x11b, calling clr!RaiseTheExceptionInternalOnly
000000002f9d6e00 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d6e20 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d6e30 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d6e40 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d6e80 000007fe99a5bb9c (MethodDesc 000007fe96f70110 +0x9c Devart.Data.MySql.ao.d(System.Exception)), calling clr!IL_Throw
000000002f9d6f18 000007fef6494fb8 clr!IL_Throw+0x45, calling clr!LazyMachStateCaptureState
000000002f9d6f30 000007fe99a5bb9c (MethodDesc 000007fe96f70110 +0x9c Devart.Data.MySql.ao.d(System.Exception)), calling clr!IL_Throw
000000002f9d6f70 000007fe96d63242 (MethodDesc 000007fe96f4bb58 +0xd2 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d6fd0 000007fef64959c5 clr!ExceptionTracker::CallHandler+0xc5
000000002f9d7010 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d7070 000007fef64958c3 clr!ExceptionTracker::CallCatchHandler+0x7f, calling clr!ExceptionTracker::CallHandler
000000002f9d70d0 000007fef64933ed clr!GCCoopNoDtor::Enter+0xe, calling clr!GetThread
000000002f9d7100 000007fef64942af clr!ProcessCLRException+0x2e6, calling clr!ExceptionTracker::CallCatchHandler
000000002f9d7148 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d71b0 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d71e0 00000000775a7f0d ntdll!RtlpExecuteHandlerForUnwind+0xd
000000002f9d7210 0000000077597d4c ntdll!RtlUnwindEx+0x539, calling ntdll!RtlpExecuteHandlerForUnwind
000000002f9d72a8 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d73a0 000007fef6494f3a clr!ExceptionTracker::ProcessOSExceptionNotification+0x34a, calling clr!_security_check_cookie
000000002f9d7490 000007fe96d63137 (MethodDesc 000007fe96f4aee0 +0x2a7 Devart.Data.MySql.bi.a(Byte[], Int32, Int32)), calling (MethodDesc 000007fe96f4bb58 +0 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d7580 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d7640 000007fe98768060 (MethodDesc 000007fe988d08e8 +0x7b0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fef4ed66b0 +0 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, mscorlib]].GetResult())
000000002f9d7660 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d7690 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d76a0 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d76d0 000007fef63d6fc4 clr!EECodeInfo::Init+0x6e
000000002f9d76e0 000007fef636ef53 clr!EECodeInfo::GetFunctionEntry+0x1d
000000002f9d7718 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d7730 000007fef634755f clr!RtlVirtualUnwind_Wrapper+0x6b, calling ntdll!RtlVirtualUnwind
000000002f9d7780 000007fef63474e2 clr!Thread::VirtualUnwindNonLeafCallFrame+0x56, calling clr!RtlVirtualUnwind_Wrapper
000000002f9d77b0 000007fe96d63137 (MethodDesc 000007fe96f4aee0 +0x2a7 Devart.Data.MySql.bi.a(Byte[], Int32, Int32)), calling (MethodDesc 000007fe96f4bb58 +0 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d78b0 000007fef6494348 clr!ClrUnwindEx+0x40, calling ntdll!RtlUnwindEx
000000002f9d79e0 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d7aa0 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d7bc0 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9d7cd0 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d7d00 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d7d20 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d7d30 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d7d40 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d7d50 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d7dd0 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d7e18 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d7e90 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d7eb0 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d7eb8 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d7ee0 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d7f78 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d7fe0 000007fef63d6fc4 clr!EECodeInfo::Init+0x6e
000000002f9d7ff0 000007fef636ef53 clr!EECodeInfo::GetFunctionEntry+0x1d
000000002f9d8028 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d8040 000007fef634755f clr!RtlVirtualUnwind_Wrapper+0x6b, calling ntdll!RtlVirtualUnwind
000000002f9d8090 000007fef63474e2 clr!Thread::VirtualUnwindNonLeafCallFrame+0x56, calling clr!RtlVirtualUnwind_Wrapper
000000002f9d80c0 000007fe96d63137 (MethodDesc 000007fe96f4aee0 +0x2a7 Devart.Data.MySql.bi.a(Byte[], Int32, Int32)), calling (MethodDesc 000007fe96f4bb58 +0 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9d8180 000007fef6494f3a clr!ExceptionTracker::ProcessOSExceptionNotification+0x34a, calling clr!_security_check_cookie
000000002f9d8290 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d82b0 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9d82c0 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d82e0 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d8360 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d83a8 000007fe98768060 (MethodDesc 000007fe988d08e8 +0x7b0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fef4ed66b0 +0 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, mscorlib]].GetResult())
000000002f9d8420 000007fe98768060 (MethodDesc 000007fe988d08e8 +0x7b0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fef4ed66b0 +0 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[[System.__Canon, mscorlib]].GetResult())
000000002f9d8440 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d8470 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d8480 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d8560 000007fef6494f3a clr!ExceptionTracker::ProcessOSExceptionNotification+0x34a, calling clr!_security_check_cookie
000000002f9d8590 00000000775988c8 ntdll!RtlRaiseException+0x48, calling ntdll!RtlCaptureContext
000000002f9d85c0 0000000077598ac8 ntdll!RtlRaiseException+0x22f, calling ntdll!RtlDispatchException
000000002f9d86f0 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d88d8 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9d8950 000007fe96d6339a (MethodDesc 000007fe96f4bda8 +0xca Devart.Common.a5.c(Byte[], Int32, Int32))
000000002f9d8a30 000007fef6494f3a clr!ExceptionTracker::ProcessOSExceptionNotification+0x34a, calling clr!_security_check_cookie
000000002f9d8b00 000007fe96d6339a (MethodDesc 000007fe96f4bda8 +0xca Devart.Common.a5.c(Byte[], Int32, Int32))
000000002f9d8b80 000007fef648d85a clr!HndCreateHandle+0x112, calling clr!StressLog::LogOn
000000002f9d8bb0 000007fef63fd315 clr!BaseDomain::CreateHandle+0x15, calling clr!GetCurrentThreadHomeHeapNumber
000000002f9d8bc0 000007fef635d0ae clr!EEUnicodeStringLiteralHashTableHelper::CompareKeys+0x7f, calling clr!GCCoop::~GCCoop
000000002f9d8c00 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d8c38 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9d8c70 000007fef5086fab (MethodDesc 000007fef4cfedf0 +0x89b System.Resources.RuntimeResourceSet.GetObject(System.String, Boolean, Boolean)), calling 000007fef6342730 (stub for System.Threading.Monitor.Exit(System.Object))
000000002f9d8ca0 000007fef64952b4 clr!IsExceptionOfType+0x15, calling clr!Object::GetTrueMethodTable
000000002f9d8cd0 000007fef6495294 clr!RaiseTheExceptionInternalOnly+0x2fe, calling kernel32!RaiseExceptionStub

jleblanc
Posts: 14
Joined: Tue 19 Apr 2016 22:31

Re: Deadlock in connection pool code

Post by jleblanc » Wed 12 Oct 2016 23:26

Code: Select all

000000002f9d8cf0 000007fef5086e89 (MethodDesc 000007fef4cfedf0 +0x779 System.Resources.RuntimeResourceSet.GetObject(System.String, Boolean, Boolean)), calling (MethodDesc 000007fef4cfedf0 +0x869 System.Resources.RuntimeResourceSet.GetObject(System.String, Boolean, Boolean))
000000002f9d8d10 000007fef64a3b19 clr!LoaderAllocator::GetStringObjRefPtrFromUnicodeString+0x2e
000000002f9d8d40 000007fef64834a6 clr!Module::ResolveStringRef+0x1f6, calling clr!GCCoop::~GCCoop
000000002f9d8dd0 000007fef649508e clr!IL_Throw+0x11b, calling clr!RaiseTheExceptionInternalOnly
000000002f9d8ea0 000007fef50866ec (MethodDesc 000007fef4e32e48 +0xc System.Resources.RuntimeResourceSet.GetString(System.String, Boolean)), calling (MethodDesc 000007fef4cfedf0 +0 System.Resources.RuntimeResourceSet.GetObject(System.String, Boolean, Boolean))
000000002f9d8ef0 000007fe96d635df (MethodDesc 000007fe96f4ea80 +0xff Devart.Common.a3.b(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9d8f88 000007fef6494fb8 clr!IL_Throw+0x45, calling clr!LazyMachStateCaptureState
000000002f9d8fa0 000007fe96d635df (MethodDesc 000007fe96f4ea80 +0xff Devart.Common.a3.b(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9d8ff0 000007fef64959c5 clr!ExceptionTracker::CallHandler+0xc5
000000002f9d9090 000007fef64958c3 clr!ExceptionTracker::CallCatchHandler+0x7f, calling clr!ExceptionTracker::CallHandler
000000002f9d90f0 000007fef64933ed clr!GCCoopNoDtor::Enter+0xe, calling clr!GetThread
000000002f9d9120 000007fef64942af clr!ProcessCLRException+0x2e6, calling clr!ExceptionTracker::CallCatchHandler
000000002f9d9168 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d91d0 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d9200 00000000775a7f0d ntdll!RtlpExecuteHandlerForUnwind+0xd
000000002f9d9230 0000000077597d4c ntdll!RtlUnwindEx+0x539, calling ntdll!RtlpExecuteHandlerForUnwind
000000002f9d92c8 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d93e0 000007fef6494f3a clr!ExceptionTracker::ProcessOSExceptionNotification+0x34a, calling clr!_security_check_cookie
000000002f9d94b0 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d9660 000007fef6438984 clr!NativeImageJitManager::GetExceptionInfo+0x3c, calling clr!NativeUnwindInfoLookupTable::GetMethodDescRVA
000000002f9d96a0 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d96d0 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d9758 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9d97d0 000007fe96d6339a (MethodDesc 000007fe96f4bda8 +0xca Devart.Common.a5.c(Byte[], Int32, Int32))
000000002f9d98d0 000007fef6494348 clr!ClrUnwindEx+0x40, calling ntdll!RtlUnwindEx
000000002f9d9a00 000007fe96d6339a (MethodDesc 000007fe96f4bda8 +0xca Devart.Common.a5.c(Byte[], Int32, Int32))
000000002f9d9be0 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9d9cf0 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9d9d20 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9d9d50 000007fef6494568 clr!ExceptionTracker::GetOrCreateTracker+0x34, calling clr!GetThread
000000002f9d9d60 000007fef63cd03d clr!Thread::GetLastNormalStackAddress+0xe, calling clr!Thread::GetStackGuarantee
000000002f9d9d70 000007fef63430bb clr!DecCantAllocCount+0x3b, calling clr!ClrFlsGetBlock
000000002f9d9df0 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9d9e38 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d9eb0 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d9ed0 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9d9ed8 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9d9f00 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9d9f98 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9da068 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9da0e0 000007fe96d6339a (MethodDesc 000007fe96f4bda8 +0xca Devart.Common.a5.c(Byte[], Int32, Int32))
000000002f9da1c0 000007fef6494f3a clr!ExceptionTracker::ProcessOSExceptionNotification+0x34a, calling clr!_security_check_cookie
000000002f9da430 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9da440 000007fef6438984 clr!NativeImageJitManager::GetExceptionInfo+0x3c, calling clr!NativeUnwindInfoLookupTable::GetMethodDescRVA
000000002f9da480 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9da4b0 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9da550 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9da5b0 00000000775988c8 ntdll!RtlRaiseException+0x48, calling ntdll!RtlCaptureContext
000000002f9da5e0 0000000077598ac8 ntdll!RtlRaiseException+0x22f, calling ntdll!RtlDispatchException
000000002f9da670 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9da710 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9da990 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9dab40 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9daba0 000007fef648d85a clr!HndCreateHandle+0x112, calling clr!StressLog::LogOn
000000002f9dabd0 000007fef63fd315 clr!BaseDomain::CreateHandle+0x15, calling clr!GetCurrentThreadHomeHeapNumber
000000002f9dac20 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9dac58 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9daca0 000007fef5087014 (MethodDesc 000007fef4e33330 +0x54 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Resources.ResourceLocator, mscorlib]].TryGetValue(System.__Canon, System.Resources.ResourceLocator ByRef)), calling clr!JIT_CheckedWriteBarrier
000000002f9dacc0 000007fef64952b4 clr!IsExceptionOfType+0x15, calling clr!Object::GetTrueMethodTable
000000002f9dacf0 000007fef6495294 clr!RaiseTheExceptionInternalOnly+0x2fe, calling kernel32!RaiseExceptionStub
000000002f9dad20 000007fef64a98e4 clr!MDInternalRW::GetUserString+0x4c
000000002f9dad60 000007fef64834a6 clr!Module::ResolveStringRef+0x1f6, calling clr!GCCoop::~GCCoop
000000002f9dadf0 000007fef649508e clr!IL_Throw+0x11b, calling clr!RaiseTheExceptionInternalOnly
000000002f9dae80 000007fef50b6709 (MethodDesc 000007fef4c09458 +0x29 System.Globalization.CultureInfo.GetFormat(System.Type)), calling clr!JIT_GetRuntimeType
000000002f9daea0 000007fef50866ec (MethodDesc 000007fef4e32e48 +0xc System.Resources.RuntimeResourceSet.GetString(System.String, Boolean)), calling (MethodDesc 000007fef4cfedf0 +0 System.Resources.RuntimeResourceSet.GetObject(System.String, Boolean, Boolean))
000000002f9daec0 000007fef50b643d (MethodDesc 000007fef4c0c530 +0x12d System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])), calling (MethodDesc 000007fef4c0c310 +0 System.Text.StringBuilder.Append(Char))
000000002f9daed0 000007fef506a981 (MethodDesc 000007fef4c09300 +0x21 System.Globalization.CultureInfo.get_UserDefaultCulture()), calling clr!JIT_GetSharedGCStaticBase_InlineGetAppDomain
000000002f9daf10 000007fef45fc86d (MethodDesc 000007fef3b747b0 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9dafa8 000007fef6494fb8 clr!IL_Throw+0x45, calling clr!LazyMachStateCaptureState
000000002f9dafc0 000007fef45fc86d (MethodDesc 000007fef3b747b0 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9db030 000007fef64959c5 clr!ExceptionTracker::CallHandler+0xc5
000000002f9db0d0 000007fef64958c3 clr!ExceptionTracker::CallCatchHandler+0x7f, calling clr!ExceptionTracker::CallHandler
000000002f9db130 000007fef64933ed clr!GCCoopNoDtor::Enter+0xe, calling clr!GetThread
000000002f9db160 000007fef64942af clr!ProcessCLRException+0x2e6, calling clr!ExceptionTracker::CallCatchHandler
000000002f9db1a8 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9db1b0 00000000775a7cce ntdll!RtlLookupFunctionTable+0xaa, calling ntdll!RtlReleaseSRWLockShared
000000002f9db210 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9db240 00000000775a7f0d ntdll!RtlpExecuteHandlerForUnwind+0xd
000000002f9db270 0000000077597d4c ntdll!RtlUnwindEx+0x539, calling ntdll!RtlpExecuteHandlerForUnwind
000000002f9db308 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9db4f0 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9db810 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9db910 000007fef6494348 clr!ClrUnwindEx+0x40, calling ntdll!RtlUnwindEx
000000002f9dba40 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9dbc20 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9dbcf0 000007fef640b9d8 clr!Thread::SafeSetThrowables+0x74, calling clr!CLRException::HandlerState::CleanupTry
000000002f9dbd30 000007fef6343a6a clr!EEHeapAllocInProcessHeap+0x46, calling ntdll!RtlAllocateHeap
000000002f9dbd60 000007fef649397e clr!StackTraceInfo::AllocateStackTrace+0x2e, calling clr!operator new
000000002f9dbd80 000007fef649391e clr!GetEHTrackerForException+0x13, calling clr!GetThread
000000002f9dbd90 000007fef64954f7 clr!ExceptionTracker::GetOrCreateTracker+0x230, calling clr!StackTraceInfo::AllocateStackTrace
000000002f9dbdb0 000007fef6495f16 clr!CEHelper::IsProcessCorruptedStateException+0x27, calling clr!GetEHTrackerForException
000000002f9dbde0 000007fef6495605 clr!CEHelper::SetupCorruptionSeverityForActiveException+0xd1, calling clr!CEHelper::IsProcessCorruptedStateException
000000002f9dbe30 000007fef6494304 clr!ProcessCLRException+0x2b2, calling clr!ClrUnwindEx
000000002f9dbe78 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9dbe80 00000000775a7cce ntdll!RtlLookupFunctionTable+0xaa, calling ntdll!RtlReleaseSRWLockShared
000000002f9dbec0 000007fef626954f MSVCR120_CLR0400!_CxxFrameHandler+0x77, calling MSVCR120_CLR0400!_InternalCxxFrameHandler
000000002f9dbef0 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9dbf10 00000000775a7e8d ntdll!RtlpExecuteHandlerForException+0xd
000000002f9dbf40 00000000775984cf ntdll!RtlDispatchException+0x45a, calling ntdll!RtlpExecuteHandlerForException
000000002f9dbf70 00000000775a719a ntdll!LdrLoadAlternateResourceModuleEx+0x55a, calling ntdll!_security_check_cookie
000000002f9dbfd8 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9dc120 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9dc2f0 00000000775cdb18 ntdll!RtlAllocateHeap+0xe4, calling ntdll!RtlpLowFragHeapAllocFromContext
000000002f9dc300 00000000775a335c ntdll!RtlFormatMessageEx+0x34c, calling ntdll!_security_check_cookie
000000002f9dc340 00000000775a6b24 ntdll!RtlpQueryInformationActivationContextBasicInformation+0x44, calling ntdll!RtlAddRefActivationContext
000000002f9dc400 00000000775cec38 ntdll!CsrClientCallServer+0x84, calling ntdll!ZwRequestWaitReplyPort
000000002f9dc430 000007fefd54af95 KERNELBASE!CreateRemoteThreadEx+0x3c7, calling KERNELBASE!_security_check_cookie
000000002f9dc570 0000000077373687 kernel32!LocalBaseRegOpenKey+0x276, calling kernel32!_security_check_cookie
000000002f9dc5f0 00000000775988c8 ntdll!RtlRaiseException+0x48, calling ntdll!RtlCaptureContext
000000002f9dc620 0000000077598ac8 ntdll!RtlRaiseException+0x22f, calling ntdll!RtlDispatchException
000000002f9dc6b0 00000000775a8553 ntdll!LdrpAccessResourceDataNoMultipleLanguage+0x13f, calling ntdll!RtlImageNtHeaderEx
000000002f9dc750 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9dc900 00000000775a8553 ntdll!LdrpAccessResourceDataNoMultipleLanguage+0x13f, calling ntdll!RtlImageNtHeaderEx
000000002f9dc990 000007fef63443fa clr!Alloc+0x54, calling clr!GetThread
000000002f9dc9c0 000007fef635a0cc clr!FastAllocatePrimitiveArray+0xd4, calling clr!Alloc
000000002f9dca30 000000007737508e kernel32!BasepNotifyLoadStringResource+0x6a, calling 000000000059004e
000000002f9dca40 000007fef6359ffd clr!AllocatePrimitiveArray+0x29, calling clr!TypeDesc::GetMethodTable
000000002f9dca50 000007fef63aa3d2 clr!StringBufferObject::ReplaceBuffer+0x56, calling MSVCR120_CLR0400!memcpy
000000002f9dca80 000007fef63aa369 clr!StringBufferObject::ReplaceBuffer+0x59, calling clr!StringBufferObject::ReplaceBuffer
000000002f9dcab0 000007fef63aa2d1 clr!COMStringBuffer::ReplaceBufferInternal+0xd5, calling clr!HelperMethodFrameRestoreState
000000002f9dcbe0 000007fef648d85a clr!HndCreateHandle+0x112, calling clr!StressLog::LogOn
000000002f9dcc10 000007fef63fd315 clr!BaseDomain::CreateHandle+0x15, calling clr!GetCurrentThreadHomeHeapNumber
000000002f9dcc20 000007fef3e35d6c (MethodDesc 000007fef3b8ebd8 +0x28c DomainBoundILStubClass.IL_STUB_PInvoke(Int32, IntPtr, UInt32, Int32, System.Text.StringBuilder, Int32, IntPtr[])), calling 000007fef63aa1fc (stub for System.Text.StringBuilder.ReplaceBufferInternal(Char*, Int32))
000000002f9dcc60 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9dcc98 000007fefd55a06d KERNELBASE!RaiseException+0x39, calling ntdll!RtlRaiseException
000000002f9dcd00 000007fef64952b4 clr!IsExceptionOfType+0x15, calling clr!Object::GetTrueMethodTable
000000002f9dcd30 000007fef6495294 clr!RaiseTheExceptionInternalOnly+0x2fe, calling kernel32!RaiseExceptionStub
000000002f9dce30 000007fef649508e clr!IL_Throw+0x11b, calling clr!RaiseTheExceptionInternalOnly
000000002f9dcf50 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9dcfe8 000007fef6494fb8 clr!IL_Throw+0x45, calling clr!LazyMachStateCaptureState
000000002f9dd000 000007fef3df4639 (MethodDesc 000007fef3b747b0 +0xf9 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling clr!IL_Throw
000000002f9dd080 000007fe96d63523 (MethodDesc 000007fe96f4ea80 +0x43 Devart.Common.a3.b(Byte[], Int32, Int32))
000000002f9dd090 000007fef3df3d87 (MethodDesc 000007fef3b83f60 +0xf7 System.Net.Sockets.Socket.Send(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)), calling 000007fef3d29e70 (stub for System.Net.UnsafeNclNativeMethods+OSSOCK.send(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags))
000000002f9dd0e0 000007fe96d6339a (MethodDesc 000007fe96f4bda8 +0xca Devart.Common.a5.c(Byte[], Int32, Int32))
000000002f9dd150 000007fe96d631f0 (MethodDesc 000007fe96f4bb58 +0x80 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9dd1c0 000007fe96d63137 (MethodDesc 000007fe96f4aee0 +0x2a7 Devart.Data.MySql.bi.a(Byte[], Int32, Int32)), calling (MethodDesc 000007fe96f4bb58 +0 Devart.Common.t.d(Byte[], Int32, Int32))
000000002f9dd230 000007fe96d62d62 (MethodDesc 000007fe96f4ac00 +0x22 Devart.Data.MySql.bi.p()), calling (MethodDesc 000007fe96f4aee0 +0 Devart.Data.MySql.bi.a(Byte[], Int32, Int32))
000000002f9dd238 000007fef64690a7 clr!ArrayNative::ArrayClear+0x73, calling clr!LazyMachStateCaptureState
000000002f9dd250 000007fef509b6d0 (MethodDesc 000007fef4cf3de0 +0x10 System.IO.MemoryStream.EnsureWriteable())
000000002f9dd280 000007fe96d62c99 (MethodDesc 000007fe96f4ac20 +0x29 Devart.Data.MySql.bi.i()), calling (MethodDesc 000007fe96f4ac00 +0 Devart.Data.MySql.bi.p())
000000002f9dd2a0 000007fe96f91e2e (MethodDesc 000007fe96f4abd0 +0xae Devart.Data.MySql.bi.z()), calling (MethodDesc 000007fe96f4aed0 +0 Devart.Data.MySql.bi.f(Byte[], Int32, Int32))
000000002f9dd2d0 000007fe96f9af7b (MethodDesc 000007fe96f47458 +0x3b Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)), calling (MethodDesc 000007fe96f4ac20 +0 Devart.Data.MySql.bi.i())
000000002f9dd2e0 000007fe96f93610 (MethodDesc 000007fe96f47448 +0xc0 Devart.Data.MySql.ae.a(Devart.Data.MySql.a, Byte[], Int32, Boolean)), calling (MethodDesc 000007fe96f4abd0 +0 Devart.Data.MySql.bi.z())
000000002f9dd2f0 000007fe96d61e8a (MethodDesc 000007fe96f0e480 +0x4a Devart.Data.MySql.au.b(Devart.Data.MySql.ba)), calling 000007fef6342730 (stub for System.Threading.Monitor.Exit(System.Object))
000000002f9dd340 000007fe96f9aea9 (MethodDesc 000007fe96f474a8 +0xf9 Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)), calling (MethodDesc 000007fe96f47458 +0 Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean))
000000002f9dd3b0 000007fe96f97247 (MethodDesc 000007fe96f76b50 +0xf7 Devart.Data.MySql.g.e())
000000002f9dd3d0 000007fe96f96ed2 (MethodDesc 000007fe96f42f48 +0xd2 Devart.Data.MySql.MySqlCommand.e()), calling clr!JIT_IsInstanceOfInterface
000000002f9dd430 000007fe96f9620c (MethodDesc 000007fe96f42f08 +0x34c Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)), calling (MethodDesc 000007fe96f76b40 +0 Devart.Data.MySql.g.m())
000000002f9dd440 000007fe96f95a2e (MethodDesc 000007fe96f42f00 +0x14e Devart.Data.MySql.MySqlCommand.InternalPrepare(Boolean, Int32, Int32)), calling (MethodDesc 000007fe96f76ac0 +0 Devart.Data.MySql.g..ctor(Devart.Data.MySql.au, System.String, Devart.Data.MySql.aq))
000000002f9dd4e0 000007fe96f94fd1 (MethodDesc 000007fe96f42788 +0x151 Devart.Common.DbCommandBase.ExecuteDbDataReader(System.Data.CommandBehavior, Boolean))
000000002f9dd500 000007fe975d9e9e (MethodDesc 000007fe96f78c78 +0x10e Devart.Common.DbParameterBaseCollection.Add(System.Object))
000000002f9dd580 000007fe96f94d9b (MethodDesc 000007fe96f42660 +0x1b Devart.Common.DbCommandBase.ExecuteNonQuery()), calling 000007fe96eb5a80
000000002f9dd5d0 000007fe98b479ce (MethodDesc 000007fe98b39300 +0x1fe TSO.Metrics.Metrics.Persistence.ActiveUserCountMetricInstance.Persist(Int32, System.Data.IDbConnection))
000000002f9dd680 000007fe98b47516 (MethodDesc 000007fe976bd100 +0x176 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector.SaveMetrics(System.Collections.Generic.IEnumerable`1<TSO.Metrics.Metrics.IMetricInstance>))
000000002f9dd760 000007fe9876827b (MethodDesc 000007fe988d08e8 +0x9cb ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector+<Collect>d__10.MoveNext()), calling (MethodDesc 000007fe976bd100 +0 ECzar.EnergyCore.ServerMetrics.ServerMetricsCollector.SaveMetrics(System.Collections.Generic.IEnumerable`1<TSO.Metrics.Metrics.IMetricInstance>))
000000002f9dd830 000007fe9972ed5a (MethodDesc 000007fe988de260 +0xda TSO.Metrics.Metrics.SNMP.Memory.MemoryFromHrStorageTableMetricTranslator.Translate(TSO.SNMP.SnmpCollectionResult)), calling (MethodDesc 000007fe988dee80 +0 TSO.SNMP.SnmpCollectionResult.get_Success())
000000002f9dd910 000007fef50cc82c (MethodDesc 000007fef4c152a0 +0x1fc System.TimeZoneInfo.GetIsDaylightSavingsFromUtc(System.DateTime, Int32, System.TimeSpan, AdjustmentRule, Boolean ByRef)), calling (MethodDesc 000007fef4c147e0 +0 System.DateTime.GetDatePart(Int32))
000000002f9dda20 000007fef509d534 (MethodDesc 000007fef4c15170 +0x114 System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc(System.DateTime, Boolean ByRef)), calling (MethodDesc 000007fef4c10d70 +0 System.TimeSpan.Add(System.TimeSpan))
000000002f9dda50 000007fe9972e4ff (MethodDesc 000007fe98ff4fb8 +0xaf TSO.Metrics.Metrics.SNMP.ActiveUserCount.WinActiveUserCountMetricTranslator+<>c.<Translate>b__1_0(TSO.SNMP.Variables.OctetString)), calling (MethodDesc 000007fef4c0b2c0 +0 System.String.Equals(System.String, System.String))
000000002f9dda60 000007fef6345a1f clr!SystemNative::__GetSystemTimeAsFileTime+0xf, calling kernel32!GetSystemTimeAsFileTimeStub
000000002f9ddaa0 000007fef509d415 (MethodDesc 000007fef4c14898 +0xb5 System.DateTime.get_Now()), calling (MethodDesc 000007fef4c14488 +0 System.DateTime..ctor(Int64, System.DateTimeKind, Boolean))
000000002f9ddac0 000007fef509e6d4 (MethodDesc 000007fef4cf47a8 +0x94 System.Threading.ExecutionContext.EstablishCopyOnWriteScope(System.Threading.Thread, Boolean, System.Threading.ExecutionContextSwitcher ByRef)), calling clr!JIT_CheckedWriteBarrier
000000002f9ddae0 000007fe98b45a4c (MethodDesc 000007fe989809a0 +0x3c TSO.Metrics.Metrics.Persistence.MetricInstanceBase.set_MeasureDateTime(System.DateTime)), calling clr!JIT_MemCpy
000000002f9ddb00 000007fef511d133 (MethodDesc 000007fef4e29df0 +0x13 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)), calling clr!JIT_ChkCastInterface
000000002f9ddb30 000007fef50a39a5 (MethodDesc 000007fef4cf4790 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9ddb50 000007fe98b4596a (MethodDesc 000007fe98b392c0 +0x3a TSO.Metrics.Metrics.Persistence.ActiveUserCountMetricInstance..ctor(UInt32)), calling (MethodDesc 000007fe98b392d8 +0 TSO.Metrics.Metrics.Persistence.ActiveUserCountMetricInstance.set_IsValid(Boolean))
000000002f9ddbd0 000007fe9972e8ce (MethodDesc 000007fe988db6d0 +0x5e TSO.Metrics.Metrics.SNMP.CPU.WinCpuMetricTranslator.Translate(TSO.SNMP.SnmpCollectionResult)), calling (MethodDesc 000007fe988dee80 +0 TSO.SNMP.SnmpCollectionResult.get_Success())
000000002f9ddc90 000007fef50a3719 (MethodDesc 000007fef4cf4780 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cf4790 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9ddca0 000007fef50ab254 (MethodDesc 000007fef4ceb9a8 +0x14 System.Environment.get_IsWinRTSupported()), calling clr!JIT_GetSharedNonGCStaticBaseNoCtor_InlineGetAppDomain
000000002f9ddcc0 000007fef5118904 (MethodDesc 000007fef4cf85d8 +0xa4 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()), calling (MethodDesc 000007fef4cf4780 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9ddcd0 000007fef511de78 (MethodDesc 000007fef4dda9c8 +0x88 System.Threading.Tasks.AwaitTaskContinuation.get_IsValidLocationForInlining()), calling clr!JIT_GetSharedGCThreadStaticBase
000000002f9ddd10 000007fef511ddce (MethodDesc 000007fef4dda9f0 +0x6e System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9ddd30 000007fef634a939 clr!JIT_GetSharedGCThreadStaticBase+0x99, calling clr!_security_check_cookie
000000002f9ddd70 000007fef5078878 (MethodDesc 000007fef4cf8e38 +0x108 System.Threading.Tasks.Task.FinishContinuations()), calling (MethodDesc 000007fef4dda9f0 +0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9dde00 000007fef50786f3 (MethodDesc 000007fef4ce8c10 +0x83 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon)), calling (MethodDesc 000007fef4cf8c80 +0 System.Threading.Tasks.Task.FinishStageThree())
000000002f9dde50 000007fef5117964 (MethodDesc 000007fef4ce0de0 +0x194 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].SetResult(System.__Canon)), calling (MethodDesc 000007fef4ce8c10 +0 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon))
000000002f9ddea0 000007fe98902477 (MethodDesc 000007fe989658d0 +0x1037 TSO.SNMP.TsoSnmpService+<GetResponseAsync>d__11.MoveNext()), calling (MethodDesc 000007fe989658d0 +0x12ec TSO.SNMP.TsoSnmpService+<GetResponseAsync>d__11.MoveNext())
000000002f9ddee0 000007fe988f3854 (MethodDesc 000007fe988deb70 +0x974 ECzar.EnergyCore.ServerMetrics.Strategies.SNMP.SnmpMetricstrategyBase+<Collect>d__12.MoveNext()), calling (MethodDesc 000007fef4ce0de0 +0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].SetResult(System.__Canon))
000000002f9de060 000007fef6345f4f clr!JIT_NewArr1+0x2bf, calling clr!HelperMethodFrameRestoreState
000000002f9de0a0 000007fef509e6d4 (MethodDesc 000007fef4cf47a8 +0x94 System.Threading.ExecutionContext.EstablishCopyOnWriteScope(System.Threading.Thread, Boolean, System.Threading.ExecutionContextSwitcher ByRef)), calling clr!JIT_CheckedWriteBarrier
000000002f9de110 000007fef50a39a5 (MethodDesc 000007fef4cf4790 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9de270 000007fef50a3719 (MethodDesc 000007fef4cf4780 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cf4790 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9de280 000007fef50ab254 (MethodDesc 000007fef4ceb9a8 +0x14 System.Environment.get_IsWinRTSupported()), calling clr!JIT_GetSharedNonGCStaticBaseNoCtor_InlineGetAppDomain
000000002f9de2a0 000007fef5118904 (MethodDesc 000007fef4cf85d8 +0xa4 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()), calling (MethodDesc 000007fef4cf4780 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9de2b0 000007fef511de78 (MethodDesc 000007fef4dda9c8 +0x88 System.Threading.Tasks.AwaitTaskContinuation.get_IsValidLocationForInlining()), calling clr!JIT_GetSharedGCThreadStaticBase
000000002f9de2f0 000007fef511ddce (MethodDesc 000007fef4dda9f0 +0x6e System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9de310 000007fef634a939 clr!JIT_GetSharedGCThreadStaticBase+0x99, calling clr!_security_check_cookie
000000002f9de350 000007fef5078878 (MethodDesc 000007fef4cf8e38 +0x108 System.Threading.Tasks.Task.FinishContinuations()), calling (MethodDesc 000007fef4dda9f0 +0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9de3e0 000007fef50786f3 (MethodDesc 000007fef4ce8c10 +0x83 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon)), calling (MethodDesc 000007fef4cf8c80 +0 System.Threading.Tasks.Task.FinishStageThree())
000000002f9de400 000007fe98b49aa2 (MethodDesc 000007fe988df000 +0x2d2 TSO.SNMP.SnmpCollectionResult.Merge(TSO.SNMP.SnmpCollectionResult)), calling (MethodDesc 000007fef4c0e088 +0 System.Collections.Generic.List`1[[System.__Canon, mscorlib]].AddRange(System.Collections.Generic.IEnumerable`1<System.__Canon>))
000000002f9de430 000007fef5117964 (MethodDesc 000007fef4ce0de0 +0x194 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].SetResult(System.__Canon)), calling (MethodDesc 000007fef4ce8c10 +0 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon))
000000002f9de4c0 000007fe988f56d6 (MethodDesc 000007fe988df850 +0xaf6 ECzar.EnergyCore.ServerMetrics.Strategies.SNMP.SnmpMetricstrategyBase+<CollectSnmpV2>d__5.MoveNext()), calling (MethodDesc 000007fef4ce0de0 +0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].SetResult(System.__Canon))
000000002f9de4e0 000007fef511ddce (MethodDesc 000007fef4dda9f0 +0x6e System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9de5d0 000007fef50786f3 (MethodDesc 000007fef4ce8c10 +0x83 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon)), calling (MethodDesc 000007fef4cf8c80 +0 System.Threading.Tasks.Task.FinishStageThree())
000000002f9de620 000007fef5117964 (MethodDesc 000007fef4ce0de0 +0x194 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].SetResult(System.__Canon)), calling (MethodDesc 000007fef4ce8c10 +0 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon))
000000002f9de650 000007fefd5410ac KERNELBASE!WaitForSingleObjectEx+0x79, calling ntdll!ZwWaitForSingleObject
000000002f9de6b0 000007fe9890384d (MethodDesc 000007fe98966b98 +0x42d TSO.SNMP.TsoSnmpService+<Resolve>d__17.MoveNext()), calling (MethodDesc 000007fef4ce0de0 +0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].SetResult(System.__Canon))
000000002f9de720 000007fef509e6d4 (MethodDesc 000007fef4cf47a8 +0x94 System.Threading.ExecutionContext.EstablishCopyOnWriteScope(System.Threading.Thread, Boolean, System.Threading.ExecutionContextSwitcher ByRef)), calling clr!JIT_CheckedWriteBarrier
000000002f9de790 000007fef50a39a5 (MethodDesc 000007fef4cf4790 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9de7b0 000007fef50a39b9 (MethodDesc 000007fef4cf4790 +0x299 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cfd490 +0 System.Threading.ExecutionContextSwitcher.Undo(System.Threading.Thread))
000000002f9de8f0 000007fef50a3719 (MethodDesc 000007fef4cf4780 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cf4790 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9de900 000007fef50ab254 (MethodDesc 000007fef4ceb9a8 +0x14 System.Environment.get_IsWinRTSupported()), calling clr!JIT_GetSharedNonGCStaticBaseNoCtor_InlineGetAppDomain
000000002f9de920 000007fef5118904 (MethodDesc 000007fef4cf85d8 +0xa4 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()), calling (MethodDesc 000007fef4cf4780 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9de930 000007fef511deb5 (MethodDesc 000007fef4dda9c8 +0xc5 System.Threading.Tasks.AwaitTaskContinuation.get_IsValidLocationForInlining()), calling clr!JIT_GetSharedGCStaticBase_InlineGetAppDomain
000000002f9de970 000007fef511ddce (MethodDesc 000007fef4dda9f0 +0x6e System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9de990 000007fef634a939 clr!JIT_GetSharedGCThreadStaticBase+0x99, calling clr!_security_check_cookie
000000002f9de9d0 000007fef5078878 (MethodDesc 000007fef4cf8e38 +0x108 System.Threading.Tasks.Task.FinishContinuations()), calling (MethodDesc 000007fef4dda9f0 +0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef))
000000002f9dea60 000007fef50786f3 (MethodDesc 000007fef4ce8c10 +0x83 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon)), calling (MethodDesc 000007fef4cf8c80 +0 System.Threading.Tasks.Task.FinishStageThree())
000000002f9dea80 000007fef50786f3 (MethodDesc 000007fef4ce8c10 +0x83 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon)), calling (MethodDesc 000007fef4cf8c80 +0 System.Threading.Tasks.Task.FinishStageThree())
000000002f9dea90 000007fef435ce6d (MethodDesc 000007fef3b72118 +0x9d System.Net.Dns.EndGetHostAddresses(System.IAsyncResult)), calling (MethodDesc 000007fef3b722a8 +0 System.Net.Dns.HostResolutionEndHelper(System.IAsyncResult))
000000002f9deab0 000007fef50785fd (MethodDesc 000007fef4ce8d10 +0x1d System.Threading.Tasks.TaskCompletionSource`1[[System.__Canon, mscorlib]].SetResult(System.__Canon)), calling (MethodDesc 000007fef4ce8c10 +0 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon))
000000002f9dead0 000007fef598d1c7 (MethodDesc 000007fef4ddcf20 +0x277 System.Threading.Tasks.TaskFactory`1[[System.__Canon, mscorlib]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2<System.IAsyncResult,System.__Canon>, System.Action`1<System.IAsyncResult>, System.Threading.Tasks.Task`1<System.__Canon>, Boolean)), calling (MethodDesc 000007fef4ce8c10 +0 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].TrySetResult(System.__Canon))
000000002f9deb00 000007fe9891d90f (MethodDesc 000007fe9894a678 +0x39f TSO.SNMP.TsoSnmpService+<>c__DisplayClass20_0`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].<DoWithRetryAsync>b__0(System.Threading.Tasks.Task`1<System.__Canon>)), calling (MethodDesc 000007fef4ce8d10 +0 System.Threading.Tasks.TaskCompletionSource`1[[System.__Canon, mscorlib]].SetResult(System.__Canon))
000000002f9deb80 000007fef5b1f49b (MethodDesc 000007fef4ed6918 +0x7b System.Threading.Tasks.TaskFactory`1+<>c__DisplayClassf`1[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].<FromAsyncImpl>b__b(System.IAsyncResult)), calling (MethodDesc 000007fef4ddcf20 +0 System.Threading.Tasks.TaskFactory`1[[System.__Canon, mscorlib]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2<System.IAsyncResult,System.__Canon>, System.Action`1<System.IAsyncResult>, System.Threading.Tasks.Task`1<System.__Canon>, Boolean))
000000002f9deba0 000007fef635fd4d clr!JIT_IsInstanceOfAny+0x2d, calling clr!ObjIsInstanceOfNoGC
000000002f9debd0 000007fef5119d67 (MethodDesc 000007fef4ce10d8 +0x57 System.Threading.Tasks.ContinuationTaskFromResultTask`1[[System.__Canon, mscorlib]].InnerInvoke()), calling clr!JIT_IsInstanceOfAny
000000002f9debf0 000007fef50a3b48 (MethodDesc 000007fef4cfd490 +0xa8 System.Threading.ExecutionContextSwitcher.Undo(System.Threading.Thread)), calling (MethodDesc 000007fef4c082a0 +0 System.Security.SecurityContext.RestoreCurrentWI(Reader, Reader, System.Security.Principal.WindowsIdentity, Boolean))
000000002f9dec20 000007fef511611e (MethodDesc 000007fef4cf94e0 +0x6e System.Threading.Tasks.Task.Execute())
000000002f9dec90 000007fef50a39a5 (MethodDesc 000007fef4cf4790 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9dedf0 000007fef50a3719 (MethodDesc 000007fef4cf4780 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 000007fef4cf4790 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9dee20 000007fef51163f5 (MethodDesc 000007fef4cf9500 +0x2c5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 000007fef4cf4780 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
000000002f9dee70 000007fef506d849 (MethodDesc 000007fef4c143f0 +0x9 System.Random.Sample()), calling (MethodDesc 000007fef4c143f8 +0 System.Random.InternalSample())
000000002f9dee80 000007fef5167dca (MethodDesc 000007fef4d01f40 +0x5a DomainNeutralILStubClass.IL_STUB_PInvoke())
000000002f9deec0 000007fef634a939 clr!JIT_GetSharedGCThreadStaticBase+0x99, calling clr!_security_check_cookie
000000002f9def00 000007fef5115a95 (MethodDesc 000007fef4cf8d10 +0x85 System.Threading.Tasks.Task.ExecuteEntry(Boolean)), calling (MethodDesc 000007fef4cf9500 +0 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef))
000000002f9def40 000007fef50d136a (MethodDesc 000007fef4de6588 +0x1ea System.Threading.ThreadPoolWorkQueue.Dispatch()), calling 000007fe96ce6aa0
000000002f9df000 000007fef634a7f3 clr!CallDescrWorkerInternal+0x83
000000002f9df018 000007fef634ad87 clr!MethodDescCallSite::CallTargetWorker+0xa1, calling clr!_chkstk
000000002f9df040 000007fef634a6de clr!CallDescrWorkerWithHandler+0x4a, calling clr!CallDescrWorkerInternal
000000002f9df080 000007fef634ae76 clr!MethodDescCallSite::CallTargetWorker+0x251, calling clr!CallDescrWorkerWithHandler
000000002f9df0c0 000007fef634ae89 clr!MethodDescCallSite::CallTargetWorker+0x264, calling clr!_security_check_cookie
000000002f9df110 000007fef634ad87 clr!MethodDescCallSite::CallTargetWorker+0xa1, calling clr!_chkstk
000000002f9df130 000007fef6349879 clr!ArgIteratorTemplate<ArgIteratorBase>::ForceSigWalk+0x16f, calling clr!ArgIteratorTemplate<ArgIteratorBase>::ComputeReturnFlags
000000002f9df150 000007fef634ad87 clr!MethodDescCallSite::CallTargetWorker+0xa1, calling clr!_chkstk
000000002f9df170 000007fef6343605 clr!MetaSig::Init+0xc4, calling clr!SigParser::SkipExactlyOne
000000002f9df230 000007fef63d0221 clr!QueueUserWorkItemManagedCallback+0x2a, calling clr!MethodDescCallSite::CallTargetWorker
000000002f9df310 000007fef634c1ad clr!`ManagedThreadBase_DispatchMiddle'::`20'::Cleanup::~Cleanup+0x3e, calling clr!GCCoop::~GCCoop
000000002f9df320 000007fef634c121 clr!ManagedThreadBase_DispatchInner+0x2d
000000002f9df350 000007fef634c1ad clr!`ManagedThreadBase_DispatchMiddle'::`20'::Cleanup::~Cleanup+0x3e, calling clr!GCCoop::~GCCoop
000000002f9df360 000007fef634c0a8 clr!ManagedThreadBase_DispatchMiddle+0x6c, calling clr!ManagedThreadBase_DispatchInner
000000002f9df3a0 000007fef634c0c1 clr!ManagedThreadBase_DispatchMiddle+0x85, calling clr!CLRException::HandlerState::CleanupTry
000000002f9df3c0 000007fef634b9da clr!CLRConfig::GetConfigValue+0xf6, calling clr!_security_check_cookie
000000002f9df400 000007fef63445f1 clr!Frame::Push+0xe, calling clr!GetThread
000000002f9df430 000007fef634af59 clr!FrameWithCookie<DebuggerU2MCatchHandlerFrame>::FrameWithCookie<DebuggerU2MCatchHandlerFrame>+0x26, calling clr!Frame::Push
000000002f9df460 000007fef634c019 clr!ManagedThreadBase_DispatchOuter+0x75, calling clr!ManagedThreadBase_DispatchMiddle
000000002f9df470 000007fef63445d5 clr!Frame::Pop+0xe, calling clr!GetThread
000000002f9df4a0 000007fef634c025 clr!ManagedThreadBase_DispatchOuter+0xc8, calling clr!Frame::Pop
000000002f9df4f0 000007fef634c15f clr!ManagedThreadBase_FullTransitionWithAD+0x2f, calling clr!ManagedThreadBase_DispatchOuter
000000002f9df550 000007fef63d01ae clr!ManagedPerAppDomainTPCount::DispatchWorkItem+0xa2, calling clr!ManagedThreadBase_FullTransitionWithAD
000000002f9df570 000007fefd5410ac KERNELBASE!WaitForSingleObjectEx+0x79, calling ntdll!ZwWaitForSingleObject
000000002f9df580 000007fef63491eb clr!CLREventWaitHelper2+0x38, calling kernel32!WaitForSingleObjectExImplementation
000000002f9df590 000007fef6349197 clr!CLREventWaitHelper+0x1f, calling clr!CLREventWaitHelper2
000000002f9df610 000007fef63ceb8b clr!CLRSemaphore::Wait+0xa7, calling clr!CLRException::HandlerState::CleanupTry
000000002f9df680 000007fef652fe0b clr!WKS::gc_heap::wait_for_gc_done+0x50, calling clr!SVR::gc_heap::disable_preemptive
000000002f9df6a0 000007fef63cf0d3 clr!PerAppDomainTPCountList::GetAppDomainIndexForThreadpoolDispatch+0x6f
000000002f9df6e0 000007fef63cf046 clr!ThreadpoolMgr::ExecuteWorkRequest+0x46
000000002f9df710 000007fef63cef3a clr!ThreadpoolMgr::WorkerThreadStart+0xf4, calling clr!ThreadpoolMgr::ExecuteWorkRequest
000000002f9df740 000007fef6343b2d clr!EEHeapFreeInProcessHeap+0x39, calling kernel32!HeapFree
000000002f9df7a0 000007fef648fcb6 clr!Thread::intermediateThreadProc+0x7d
000000002f9df830 00000000775aa0f8 ntdll!LdrInitializeThunk+0x18, calling ntdll!NtContinue
000000002f9dfca0 000007fef648fc9f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
000000002f9dfce0 00000000773759bd kernel32!BaseThreadInitThunk+0xd
000000002f9dfd10 00000000775aa2e1 ntdll!RtlUserThreadStart+0x1d

Pinturiccio
Devart Team
Posts: 2420
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in connection pool code

Post by Pinturiccio » Mon 17 Oct 2016 15:53

Unfortunately, the provided information could not help us. Please specify the following:
1. Connection strings (roughly, without credentials) which are used in your application.
2. Describe the scenario implemented in your application. Do you use a separate connection for each thread or you synchronize threads in code and use the same connection in multiple threads? How many commands use the same connection?
3. How often does your application stop responding? Does it happen every time or only sometimes?
4. How much time passes from the start of your application till it stops responding?
5. Please also try to localize the issue and send us a small test project to reproduce the issue in our environment.

jleblanc
Posts: 14
Joined: Tue 19 Apr 2016 22:31

Re: Deadlock in connection pool code

Post by jleblanc » Mon 17 Oct 2016 16:56

1. Our connection string

Code: Select all

User Id=user;Password=password;Host=localhost;Port=3306;Database=tso_db;Unicode=True;Default Command Timeout=180;Min Pool Size=10;Max Pool Size=500;Validate Connection=True
2. We are using a separate connection per thread; threads do not share connections. We have many workers all trying to save data to the database at the same time, and when operating normally we do not maximise the dotConnect connection pool (pool size stays below 500).

3. We noticed this at a customer's site, but have only seen it this one time.

4. Unknown; our application runs 24/7.

5. I will try and replicate this in a sample application, but I anticipate that will be very difficult. In the meantime, would the memory dump of our application be helpful? It will let you examine the exact threads that are deadlocked and the locks they are holding/waiting for.

Edit: the memory dump is from a 64bit process.

jleblanc
Posts: 14
Joined: Tue 19 Apr 2016 22:31

Re: Deadlock in connection pool code

Post by jleblanc » Wed 19 Oct 2016 21:36

Unfortunately, I am not able to reproduce this issue. The best data I have to offer at this time is the memory dump of our application.

Pinturiccio
Devart Team
Posts: 2420
Joined: Wed 02 Nov 2011 09:44

Re: Deadlock in connection pool code

Post by Pinturiccio » Fri 21 Oct 2016 15:20

We need to reproduce the problem in our environment to find out the true reason. Unfortunately, in case when an issue happened just once, and can't be reproduced after this, we unable to reproduce and study it.

If you are able to localize the issue in a sample project, send it to us.

Post Reply