We were having performance issues with our database; it was extremely busy doing a lot of processing while our application was still trying to insert new data. We use connection pooling, and what happened is our connection pool maxed out and our application completely stalled.
None of our database commands were going through to MySQL and most of our apps processing completely stopped.
I took a memory dump of our application and did some investigation, and it looks like a deadlock occurred inside dotConnect's connection pooling code.
Here is a dump of all of the threads running in our application, truncated for brevity.
Code: Select all
0:252> !threads
ThreadCount: 485
UnstartedThread: 0
BackgroundThread: 473
PendingThread: 0
DeadThread: 2
Hosted Runtime: no
Lock
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
0 1 105c 0000000000434140 2a020 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA
2 2 130c 000000000043efb0 2b220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA (Finalizer)
4 3 390 0000000019342710 1020220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 Ukn (Threadpool Worker)
5 7 1f0c 00000000193739b0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000062513b0
6 8 2528 00000000193ac770 102a220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA (Threadpool Worker)
22 249 1918 00000000193dbcc0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061c6c88
23 425 14dc 000000002afdee20 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006265d60
24 403 11d8 000000001b549530 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000622faa8
25 19 1ffc 000000001b578550 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000062263e0
26 124 384 000000002bb72810 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006263dd0
27 279 1144 000000002add5ca0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005e017f0
28 137 1064 000000002ad20d00 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000762ee28
29 281 1414 000000002ad243b0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000055064c0
30 136 10ac 000000002bcff650 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006283da8
31 243 1e04 00000000193dd430 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000076578f0
32 275 e48 00000000193ddc00 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061ed820
33 203 1448 00000000193db4f0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007035e78
34 419 18cc 000000002c357e10 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000621cd98
35 247 e04 000000002aeb6870 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006276518
36 152 1e20 000000002adc1af0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000761e320
37 208 1894 000000002adc49d0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA (Threadpool Worker)
38 451 1700 000000002bb93c60 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000763f278
39 313 1f08 000000002bb91d20 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000006506378
40 270 1dc8 000000002bb8cf00 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000076474a0
41 394 b20 000000002bc10c60 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061ebae0
42 337 1418 000000002bc11c00 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000076167b0
43 402 13d8 000000002b4d8c90 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000004c78370
44 92 8b4 000000002af7d5f0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000532dc18
45 274 10cc 000000002bcd6710 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005ba3438
46 212 2354 000000002ae74ad0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007637050
47 424 23b4 000000002b025650 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005ea01c8
48 205 1768 000000002adba410 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000061ddd88
49 202 1790 000000001b63a260 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005356730
50 284 26a8 000000002bcd6ee0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000625bba8
51 11 df0 000000002ad58410 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000760e588
52 154 1a00 000000001b558590 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005d7c2d8
53 222 2050 000000001b55a4d0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007626548
54 251 21c0 000000002b3f4f20 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005e89620
55 259 1cd8 000000002b0bbe50 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000005c5bbc0
56 47 24c8 000000001b553dc0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000765fb18
57 277 276c 000000001b5e72c0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA (Threadpool Worker)
58 401 f5c 000000002c39ca70 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000063ff180
199 172 1a24 000000002b2c4cb0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007043428
200 113 1798 000000002b2c3540 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070511a8
201 311 19e4 000000002b2c1dd0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070593d0
202 458 25b0 000000002b2c2d70 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007086168
203 22 1714 000000002b2c25a0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000709cbe0
204 75 227c 000000002b2c5480 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070ba380
205 340 ecc 000000002b3f2810 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 00000000070deb20
206 339 21d8 000000002b3f37b0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007102f40
207 346 2668 000000002b3f56f0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 000000000714b090
246 343 25c 000000002ad48470 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 1 MTA (Threadpool Worker) Devart.Data.MySql.MySqlException 0000000007606360
247 54 27f8 000000002afdfdc0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA (Threadpool Worker)
248 356 25fc 000000001b6bb3a0 3029220 Preemptive 0000000000000000:0000000000000000 0000000000425490 0 MTA (Threadpool Worker)
Code: Select all
0:252> !PrintException /d 00000000061c6c88
Exception object: 00000000061c6c88
Exception type: Devart.Data.MySql.MySqlException
Message: Lost connection to MySQL server during query
InnerException: System.TimeoutException, Use !PrintException 00000000061c6ad8 to see more.
StackTrace (generated):
SP IP Function
0000000021646E10 000007FE99A5BB9C Devart_Data_MySql!Devart.Data.MySql.ao.d(System.Exception)+0x9c
0000000021646E50 000007FE96D63242 Devart_Data_MySql!Devart.Common.t.d(Byte[], Int32, Int32)+0xd2
000000002164D0A0 000007FE96D63137 Devart_Data_MySql!Devart.Data.MySql.bi.a(Byte[], Int32, Int32)+0x2a7
000000002164D110 000007FE96D62D62 Devart_Data_MySql!Devart.Data.MySql.bi.p()+0x22
000000002164D160 000007FE96D62C99 Devart_Data_MySql!Devart.Data.MySql.bi.i()+0x29
000000002164D1B0 000007FE96F9B1F4 Devart_Data_MySql!Devart.Data.MySql.ae.a(Devart.Data.MySql.aa[] ByRef, Int32 ByRef, Boolean)+0x2b4
000000002164D220 000007FE96F9AEA9 Devart_Data_MySql!Devart.Data.MySql.ae.a(Byte[], Int32, Boolean, System.String)+0xf9
000000002164D290 000007FE96F97247 Devart_Data_MySql!Devart.Data.MySql.g.e()+0xf7
000000002164D310 000007FE96F9620C Devart_Data_MySql!Devart.Data.MySql.MySqlCommand.InternalExecute(System.Data.CommandBehavior, System.IDisposable, Int32, Int32)+0x34c
StackTraceString: <none>
HResult: 80004005
Code: Select all
0:000> !dlk
Examining SyncBlocks...
Scanning for ReaderWriterLock(Slim) instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Scanning for threads waiting on SyncBlocks...
*** ERROR: Module load completed but symbols could not be loaded for Devart.Data.MySql.dll
Scanning for threads waiting on ReaderWriterLock locks...
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*** WARNING: Unable to verify checksum for System.ServiceModel.Internals.ni.dll
Scanning for threads waiting on ReaderWriterLocksSlim locks...
*** WARNING: Unable to verify checksum for System.Data.ni.dll
Scanning for threads waiting on CriticalSections...
*** WARNING: Unable to verify checksum for System.ni.dll
*** WARNING: Unable to verify checksum for System.ServiceProcess.ni.dll
WARNING: Thread 2970 exited during dump writing (open status NTSTATUS 0xC000000B)
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
*DEADLOCK DETECTED*
CLR thread 0x4c holds the lock on SyncBlock 000000002bceecd8 OBJ:000000000367ee20[System.Collections.ArrayList]
...and is waiting for the lock on SyncBlock 000000002bceec38 OBJ:0000000005e31a28[Devart.Data.MySql.MySqlInternalConnection]
CLR thread 0xcd holds the lock on SyncBlock 000000002bceec38 OBJ:0000000005e31a28[Devart.Data.MySql.MySqlInternalConnection]
...and is waiting for the lock on SyncBlock 000000002bceecd8 OBJ:000000000367ee20[System.Collections.ArrayList]
CLR Thread 0x4c is waiting at Devart.Common.DbConnectionPoolGroup.a(Devart.Common.DbConnectionPool)(+0x25 IL,+0xa2 Native)
CLR Thread 0xcd is waiting at Devart.Common.DbConnectionPool.RemoveObject(System.Object)(+0x8 IL,+0x31 Native)
1 deadlock detected.
Code: Select all
0:000> !mwaits
Examining SyncBlocks...
Scanning for ReaderWriterLock(Slim) instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Scanning for threads waiting on SyncBlocks...
Scanning for threads waiting on ReaderWriterLock locks...
Scanning for threads waiting on ReaderWriterLocksSlim locks...
Scanning for threads waiting on CriticalSections...
WARNING: Thread 2970 exited during dump writing (open status NTSTATUS 0xC000000B)
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
Scanning threads for waits on unknown locks...
WARNING: Thread 2970 exited during dump writing (open status NTSTATUS 0xC000000B)
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
WARNING: Teb 496 pointer is NULL - defaulting to 00000000`7ffde000
WARNING: 00000000`7ffde000 does not appear to be a TEB
ClrThread DbgThread OsThread LockType Lock LockLevel
------------------------------------------------------------------------------
0xf 75 0x15c SyncBlock 0000000019325cc8
0x1a2 331 0x178 SyncBlock 0000000019325cc8
0xe6 117 0x1d0 SyncBlock 000000002bceecd8
0x8a 359 0x248 SyncBlock 0000000019325cc8
0x157 246 0x25c SyncBlock 000000002bceecd8
0x7e 341 0x26c SyncBlock 0000000019325cc8
0x168 154 0x2d0 SyncBlock 000000002bceecd8
0x10a 449 0x2e0 SyncBlock 0000000019325cc8
0x6d 126 0x2f0 SyncBlock 000000002bceecd8
0x5e 444 0x308 SyncBlock 0000000019325cc8
0xb2 234 0x34c SyncBlock 000000002bceecd8
0x7c 26 0x384 SyncBlock 000000002bceecd8
0x3 4 0x390 <Unknown> 0000000000000000
0x11a 307 0x3a0 SyncBlock 0000000019325cc8
0x9e 257 0x3ac SyncBlock 0000000019325cc8
0x6b 215 0x3b4 SyncBlock 000000002bceecd8
0x108 278 0x3d0 SyncBlock 0000000019325cc8
0xe9 161 0x478 SyncBlock 000000002bceecd8
0x196 279 0x47c SyncBlock 0000000019325cc8
0x10d 472 0x484 SyncBlock 0000000019325cc8
0x135 262 0x490 SyncBlock 0000000019325cc8
0x1d4 171 0x510 SyncBlock 000000002bceecd8
0x1ac 181 0x534 SyncBlock 000000002bceecd8
0x6a 216 0x53c SyncBlock 000000002bceecd8
0x16f 108 0x560 SyncBlock 000000002bceecd8
0x1b3 260 0x594 SyncBlock 0000000019325cc8
0xd8 402 0x5ac SyncBlock 0000000019325cc8
0x1dd 478 0x5e0 SyncBlock 0000000019325cc8
0x102 421 0x5ec SyncBlock 0000000019325cc8
0x1f 363 0x5f4 SyncBlock 0000000019325cc8
0x1a7 185 0x63c SyncBlock 000000002bceecd8
0x51 132 0x640 SyncBlock 000000002bceecd8
0x54 70 0x654 SyncBlock 0000000019325cc8
0x1d3 470 0x674 SyncBlock 0000000019325cc8
0x0 10 0x680 <Unknown> 0000000000000000
0x17c 469 0x6e4 SyncBlock 0000000019325cc8
0x18c 107 0x6e8 SyncBlock 000000002bceecd8
0x81 114 0x6fc SyncBlock 000000002bceecd8
0xe7 230 0x728 SyncBlock 000000002bceecd8
0x1d1 62 0x72c SyncBlock 000000002bceecd8
0x1b6 217 0x744 SyncBlock 000000002bceecd8
0xbc 101 0x798 SyncBlock 000000002bceecd8
0xe5 94 0x7fc SyncBlock 000000002bceecd8
0xdf 263 0x810 SyncBlock 0000000019325cc8
0x19c 142 0x81c SyncBlock 000000002bceecd8
0x6e 452 0x820 SyncBlock 0000000019325cc8
0x55 447 0x824 SyncBlock 0000000019325cc8
0x172 329 0x828 SyncBlock 0000000019325cc8
0x8c 273 0xe18 SyncBlock 0000000019325cc8
0xb7 349 0xe28 SyncBlock 0000000019325cc8
0x11d 131 0xe2c SyncBlock 000000002bceecd8
0x12b 68 0xe38 SyncBlock 000000002bceecd8
0xd2 112 0xe40 SyncBlock 000000002bceecd8
0x113 32 0xe48 SyncBlock 000000002bceecd8
0x11f 425 0xe88 SyncBlock 0000000019325cc8
0xb4 265 0xe9c SyncBlock 0000000019325cc8
0x154 205 0xecc SyncBlock 000000002bceecd8
0x21 21 0xed4 <Unknown> 0000000000000000
0x129 429 0xf14 SyncBlock 0000000019325cc8
0x4e 232 0xf24 SyncBlock 000000002bceecd8
0xf2 288 0xf40 SyncBlock 0000000019325cc8
0x68 92 0xf58 SyncBlock 000000002bceecd8
0x191 58 0xf5c SyncBlock 000000002bceecd8
0x6c 474 0xfd8 SyncBlock 0000000019325cc8
0x2d 335 0x1004 SyncBlock 0000000019325cc8
0x15d 140 0x100c SyncBlock 000000002bceecd8
0x123 391 0x1010 SyncBlock 0000000019325cc8
0xa 98 0x1014 SyncBlock 000000002bceecd8
0x173 144 0x103c SyncBlock 000000002bceecd8
0x199 295 0x1050 SyncBlock 0000000019325cc8
0x1 0 0x105c <Unknown> 0000000000000000
0x89 28 0x1064 SyncBlock 000000002bceecd8
0xb0 371 0x1088 SyncBlock 0000000019325cc8
0xbb 440 0x1094 SyncBlock 0000000019325cc8
0x88 30 0x10ac SyncBlock 000000002bceecd8
0x19b 398 0x10b8 SyncBlock 0000000019325cc8
0x187 303 0x10c0 SyncBlock 0000000019325cc8
0x112 45 0x10cc SyncBlock 000000002bceecd8
0x0 9 0x10e8 <Unknown> 0000000000000000
0x56 208 0x1118 SyncBlock 000000002bceecd8
0x1c9 244 0x1130 SyncBlock 000000002bceecd8
0x117 27 0x1144 SyncBlock 000000002bceecd8
0x15e 227 0x1150 SyncBlock 000000002bceecd8
0x32 280 0x116c SyncBlock 0000000019325cc8
0x1cb 290 0x1174 SyncBlock 0000000019325cc8
0x1a5 351 0x117c SyncBlock 0000000019325cc8
0x109 175 0x1184 SyncBlock 000000002bceecd8
0x50 450 0x1188 SyncBlock 0000000019325cc8
0x16e 266 0x1198 SyncBlock 0000000019325cc8
0x142 308 0x11a8 SyncBlock 0000000019325cc8
0x1a4 59 0x11b0 SyncBlock 000000002bceecd8
0xa1 313 0x11b4 SyncBlock 0000000019325cc8
0x18b 343 0x11b8 SyncBlock 0000000019325cc8
0x193 24 0x11d8 SyncBlock 000000002bceecd8
0xc5 113 0x11e0 SyncBlock 000000002bceecd8
0xa6 323 0x11f4 SyncBlock 0000000019325cc8
0x1d0 314 0x11f8 SyncBlock 0000000019325cc8
0xee 102 0x1224 SyncBlock 000000002bceecd8
0x7d 166 0x123c SyncBlock 000000002bceecd8
0x16a 231 0x124c SyncBlock 000000002bceecd8
0xcf 264 0x1268 SyncBlock 0000000019325cc8
0x120 173 0x1280 SyncBlock 000000002bceecd8
0xd3 416 0x129c SyncBlock 0000000019325cc8
0x2a 370 0x12b4 SyncBlock 0000000019325cc8
0x16b 404 0x12bc SyncBlock 0000000019325cc8
0x5a 225 0x12f4 SyncBlock 000000002bceecd8
0x11b 379 0x1300 SyncBlock 0000000019325cc8
0x2 2 0x130c <Unknown> 0000000000000000
0x30 268 0x1334 SyncBlock 0000000019325cc8
0xf6 196 0x1348 SyncBlock 000000002bceecd8
0x1e3 481 0x1350 <Unknown> 0000000000000000
0x38 88 0x135c SyncBlock 000000002bceecd8
0xed 414 0x136c SyncBlock 0000000019325cc8
0x52 415 0x1370 SyncBlock 0000000019325cc8
0x114 282 0x1378 SyncBlock 0000000019325cc8
0x1ba 220 0x13a4 SyncBlock 000000002bceecd8
0x18e 150 0x13bc SyncBlock 000000002bceecd8
0x4c 252 0x13d4 SyncBlock 000000002bceec38
0x192 43 0x13d8 SyncBlock 000000002bceecd8
0x84 147 0x13dc SyncBlock 000000002bceecd8
0x8e 348 0x13e4 SyncBlock 0000000019325cc8
0x4a 468 0x13f0 SyncBlock 0000000019325cc8
0x24 76 0x140c SyncBlock 000000002bceecd8
0x119 29 0x1414 SyncBlock 000000002bceecd8
0x151 42 0x1418 SyncBlock 000000002bceecd8
0xcb 33 0x1448 SyncBlock 000000002bceecd8
0x4f 448 0x1450 SyncBlock 0000000019325cc8
0x5d 382 0x1460 SyncBlock 0000000019325cc8
0x194 96 0x146c SyncBlock 000000002bceecd8
0xb5 397 0x147c SyncBlock 0000000019325cc8
0x1ce 410 0x16e0 SyncBlock 0000000019325cc8
0x91 187 0x16e4 SyncBlock 000000002bceecd8
0x1c3 38 0x1700 SyncBlock 000000002bceecd8
0x16 203 0x1714 SyncBlock 000000002bceecd8
0x1b4 345 0x1750 SyncBlock 0000000019325cc8
0x13a 328 0x1764 SyncBlock 0000000019325cc8
0xcd 48 0x1768 SyncBlock 000000002bceecd8
0x13d 153 0x1770 SyncBlock 000000002bceecd8
0x26 350 0x1774 SyncBlock 0000000019325cc8
0x8d 103 0x1788 SyncBlock 000000002bceecd8
0xca 49 0x1790 SyncBlock 000000002bceecd8
0x71 200 0x1798 SyncBlock 000000002bceecd8
0xc8 310 0x179c SyncBlock 0000000019325cc8
0x10 184 0x17ac SyncBlock 000000002bceecd8
0x78 236 0x17c0 SyncBlock 000000002bceecd8
0x15c 374 0x17e8 SyncBlock 0000000019325cc8
0x195 432 0x17ec SyncBlock 0000000019325cc8
0x16d 419 0x1804 SyncBlock 0000000019325cc8
0x17 368 0x1814 SyncBlock 0000000019325cc8
0x1e6 479 0x1818 SyncBlock 0000000019325cc8
0xc7 136 0x1830 SyncBlock 000000002bceecd8
0x99 296 0x1838 SyncBlock 0000000019325cc8
0x19a 455 0x183c SyncBlock 0000000019325cc8
0x150 428 0x1840 SyncBlock 0000000019325cc8
0xa0 165 0x1844 SyncBlock 000000002bceecd8
0xbd 275 0x1848 SyncBlock 0000000019325cc8
0xfe 457 0x184c SyncBlock 0000000019325cc8
0x179 254 0x1850 SyncBlock 0000000019325cc8
0xab 462 0x1854 SyncBlock 0000000019325cc8
0xda 436 0x185c SyncBlock 0000000019325cc8
0xef 369 0x1874 SyncBlock 0000000019325cc8
0x44 241 0x188c SyncBlock 000000002bceecd8
0x133 211 0x1890 SyncBlock 000000002bceecd8
0xd0 37 0x1894 SyncBlock 0000000019325cc8
0xdc 145 0x1898 SyncBlock 000000002bceecd8
0x15f 80 0x18a0 SyncBlock 000000002bceecd8
0x5f 83 0x18ac SyncBlock 000000002bceecd8
0xd5 218 0x18bc SyncBlock 000000002bceecd8
0x1c4 77 0x18c0 SyncBlock 000000002bceecd8
0x1a3 34 0x18cc SyncBlock 000000002bceecd8
0x149 411 0x1908 SyncBlock 0000000019325cc8
0xf9 22 0x1918 SyncBlock 000000002bceecd8
0xd7 441 0x1948 SyncBlock 0000000019325cc8
0x111 180 0x1950 SyncBlock 000000002bceecd8
0x12c 309 0x1954 SyncBlock 0000000019325cc8
0x20 20 0x1964 SyncBlock 0000000019325cc8
0x1cf 430 0x19a4 SyncBlock 0000000019325cc8
0x7f 212 0x19a8 SyncBlock 000000002bceecd8
0x66 129 0x19cc SyncBlock 000000002bceecd8
0xa9 281 0x19d8 SyncBlock 0000000019325cc8
0x85 195 0x19dc SyncBlock 000000002bceecd8
0x137 201 0x19e4 SyncBlock 000000002bceecd8
0x19 16 0x19e8 <Unknown> 0000000000000000
0x6f 365 0x19f4 SyncBlock 0000000019325cc8
0x4d 394 0x19f8 SyncBlock 0000000019325cc8
0x9a 52 0x1a00 SyncBlock 000000002bceecd8
0x147 120 0x1a04 SyncBlock 000000002bceecd8
0x79 321 0x1a10 SyncBlock 0000000019325cc8
0xac 199 0x1a24 SyncBlock 000000002bceecd8
0x80 332 0x1a40 SyncBlock 0000000019325cc8
0x11e 253 0x1a50 SyncBlock 0000000019325cc8
0x5b 276 0x1a74 SyncBlock 0000000019325cc8
0x1c0 118 0x1a80 SyncBlock 000000002bceecd8
0x144 137 0x1a8c SyncBlock 000000002bceecd8
0x105 209 0x1ac0 SyncBlock 000000002bceecd8
0x69 157 0x1ac4 SyncBlock 000000002bceecd8
0x148 190 0x1ac8 SyncBlock 000000002bceecd8
0x6 116 0x1ad8 SyncBlock 000000002bceecd8
0x0 3 0x1af0 <Unknown> 0000000000000000
0x77 380 0x1af4 SyncBlock 0000000019325cc8
0x1a1 163 0x1b00 SyncBlock 000000002bceecd8
0x132 261 0x1b08 SyncBlock 0000000019325cc8
0x12f 123 0x1b24 SyncBlock 000000002bceecd8
0xc0 467 0x1b4c SyncBlock 0000000019325cc8
0x1c6 176 0x1d3c SyncBlock 000000002bceecd8
0xba 97 0x1d40 SyncBlock 000000002bceecd8
0x2b 121 0x1d44 SyncBlock 000000002bceecd8
0x1a0 298 0x1da0 SyncBlock 0000000019325cc8
0xc6 458 0x1db4 SyncBlock 0000000019325cc8
0x31 192 0x1db8 SyncBlock 000000002bceecd8
0xaf 460 0x1dc0 SyncBlock 0000000019325cc8
0x37 259 0x1dc4 SyncBlock 0000000019325cc8
0x10e 40 0x1dc8 SyncBlock 000000002bceecd8
0x14b 339 0x1df8 SyncBlock 0000000019325cc8
0xf3 31 0x1e04 SyncBlock 000000002bceecd8
0x188 93 0x1e08 SyncBlock 000000002bceecd8
0x98 36 0x1e20 SyncBlock 000000002bceecd8
0x74 322 0x1e38 SyncBlock 0000000019325cc8
0x1d 352 0x1e3c SyncBlock 0000000019325cc8
0xb6 423 0x1e44 SyncBlock 0000000019325cc8
0xa7 130 0x1e5c SyncBlock 000000002bceecd8
0x5 162 0x1e60 SyncBlock 000000002bceecd8
0x189 270 0x1e6c SyncBlock 0000000019325cc8
0x0 8 0x1e70 <Unknown> 0000000000000000
0x76 406 0x1e80 SyncBlock 0000000019325cc8
0x33 182 0x1e84 SyncBlock 000000002bceecd8
0x75 213 0x1e94 SyncBlock 000000002bceecd8
0x9d 69 0x1eb4 SyncBlock 000000002bceecd8
0x143 318 0x1ef8 SyncBlock 0000000019325cc8
0x139 39 0x1f08 SyncBlock 000000002bceecd8
0x7 5 0x1f0c SyncBlock 000000002bceecd8
0x1b5 119 0x1f10 SyncBlock 000000002bceecd8
0x1bc 143 0x1f24 SyncBlock 000000002bceecd8
0x1e8 489 0x1f34 <Unknown> 0000000000000000
0x3c 71 0x1f50 SyncBlock 000000002bceecd8
0x96 302 0x1f5c SyncBlock 0000000019325cc8
0x1d9 471 0x1f64 SyncBlock 0000000019325cc8
0xae 274 0x1f78 SyncBlock 0000000019325cc8
0x104 473 0x1f8c SyncBlock 0000000019325cc8
0x156 446 0x1fa4 SyncBlock 0000000019325cc8
0xce 272 0x1fc0 SyncBlock 0000000019325cc8
0x1c2 194 0x1fd0 SyncBlock 000000002bceecd8
0x39 442 0x1fe4 SyncBlock 0000000019325cc8
0x1ad 300 0x1ff4 SyncBlock 0000000019325cc8
0x13 25 0x1ffc SyncBlock 000000002bceecd8
0x0 12 0x2004 <Unknown> 0000000000000000
0xf5 445 0x202c SyncBlock 0000000019325cc8
0xde 53 0x2050 SyncBlock 000000002bceecd8
0x4 357 0x2068 SyncBlock 0000000019325cc8
0x3d 168 0x206c SyncBlock 000000002bceecd8
0x136 453 0x2070 SyncBlock 0000000019325cc8
0x159 367 0x2074 SyncBlock 0000000019325cc8
0x59 66 0x2080 SyncBlock 000000002bceecd8
0x9 111 0x2094 SyncBlock 000000002bceecd8
0x27 360 0x20a0 SyncBlock 0000000019325cc8
0xe1 172 0x20a4 SyncBlock 000000002bceecd8
0x130 221 0x20bc SyncBlock 000000002bceecd8
0x47 90 0x20dc SyncBlock 000000002bceecd8
0x1c1 222 0x20f4 SyncBlock 000000002bceecd8
0x14c 85 0x210c SyncBlock 000000002bceecd8
0x10f 167 0x2118 SyncBlock 000000002bceecd8
0x1be 294 0x211c SyncBlock 0000000019325cc8
0xa2 267 0x213c SyncBlock 0000000019325cc8
0x122 81 0x2150 SyncBlock 000000002bceecd8
0x43 99 0x215c SyncBlock 000000002bceecd8
0xd9 378 0x217c SyncBlock 0000000019325cc8
0x3a 324 0x2194 SyncBlock 0000000019325cc8
0xe 297 0x2198 SyncBlock 0000000019325cc8
0x10b 346 0x21a4 SyncBlock 0000000019325cc8
0xd6 284 0x21b0 SyncBlock 0000000019325cc8
0xe2 233 0x21b8 SyncBlock 000000002bceecd8
0xfb 54 0x21c0 SyncBlock 000000002bceecd8
0x13f 189 0x21c8 SyncBlock 000000002bceecd8
0x153 206 0x21d8 SyncBlock 000000002bceecd8
0x13c 256 0x2210 SyncBlock 0000000019325cc8
0x126 386 0x2230 SyncBlock 0000000019325cc8
0x29 84 0x2240 SyncBlock 000000002bceecd8
0x145 239 0x2250 SyncBlock 000000002bceecd8
0xb1 424 0x2260 SyncBlock 0000000019325cc8
0x4b 204 0x227c SyncBlock 000000002bceecd8
0x118 91 0x228c SyncBlock 000000002bceecd8
0x183 475 0x2298 SyncBlock 0000000019325cc8
0x8f 353 0x22c8 SyncBlock 0000000019325cc8
0x162 243 0x22e4 SyncBlock 000000002bceecd8
0xc 340 0x22f4 SyncBlock 0000000019325cc8
0xd1 431 0x2328 SyncBlock 0000000019325cc8
0x163 106 0x232c SyncBlock 000000002bceecd8
0x1c8 355 0x2338 SyncBlock 0000000019325cc8
0x34 229 0x2344 SyncBlock 000000002bceecd8
0x83 179 0x2348 SyncBlock 000000002bceecd8
0xd4 46 0x2354 SyncBlock 000000002bceecd8
0x1d7 361 0x2368 SyncBlock 0000000019325cc8
0x9b 396 0x237c SyncBlock 0000000019325cc8
0x14 13 0x2394 SyncBlock 000000002bceecd8
0x165 128 0x2398 SyncBlock 000000002bceecd8
0x1a8 47 0x23b4 SyncBlock 000000002bceecd8
0x15b 223 0x23b8 SyncBlock 000000002bceecd8
0x1d8 433 0x23c0 SyncBlock 0000000019325cc8
0x46 198 0x23d4 SyncBlock 000000002bceecd8
0x1da 476 0x23d8 SyncBlock 0000000019325cc8
0x17d 139 0x23e0 SyncBlock 000000002bceecd8
0x0 1 0x23e4 <Unknown> 0000000000000000
0x1c 465 0x23ec SyncBlock 0000000019325cc8
0x167 305 0x23f0 SyncBlock 0000000019325cc8
0xcc 127 0x23fc SyncBlock 000000002bceecd8
0x166 319 0x241c SyncBlock 0000000019325cc8
0x48 160 0x2420 SyncBlock 000000002bceecd8
0x1cd 292 0x244c SyncBlock 0000000019325cc8
0x16c 413 0x245c SyncBlock 0000000019325cc8
0x1aa 122 0x2460 SyncBlock 000000002bceecd8
0x9f 375 0x2478 SyncBlock 0000000019325cc8
0x25 403 0x2494 SyncBlock 0000000019325cc8
0x18f 327 0x249c SyncBlock 0000000019325cc8
0xc3 342 0x24a0 SyncBlock 0000000019325cc8
0xfa 73 0x24a4 SyncBlock 000000002bceecd8
0x42 306 0x24a8 SyncBlock 0000000019325cc8
0xbe 438 0x24ac SyncBlock 0000000019325cc8
0x2f 56 0x24c8 SyncBlock 000000002bceecd8
0x1d2 277 0x24cc SyncBlock 0000000019325cc8
0x40 286 0x24f0 SyncBlock 0000000019325cc8
0x72 427 0x2500 SyncBlock 0000000019325cc8
0x1a6 422 0x2508 SyncBlock 0000000019325cc8
0x15 14 0x2518 <Unknown> 0000000000000000
0x106 409 0x2524 SyncBlock 0000000019325cc8
0x28 347 0x2530 SyncBlock 0000000019325cc8
0x1e 19 0x2544 <Unknown> 0000000000000000
0x17e 317 0x254c SyncBlock 0000000019325cc8
0xe3 183 0x2570 SyncBlock 000000002bceecd8
0x7b 395 0x2594 SyncBlock 0000000019325cc8
0x12d 437 0x259c SyncBlock 0000000019325cc8
0x1ca 202 0x25b0 SyncBlock 000000002bceecd8
0xad 169 0x25c4 SyncBlock 000000002bceecd8
0x2e 376 0x25d8 SyncBlock 0000000019325cc8
0x171 283 0x25e8 SyncBlock 0000000019325cc8
0x121 82 0x25ec SyncBlock 000000002bceecd8
0x164 248 0x25fc SyncBlock 0000000019325cc8
0x58 237 0x2604 SyncBlock 000000002bceecd8
0xc9 224 0x2628 SyncBlock 000000002bceecd8
0x185 439 0x2630 SyncBlock 0000000019325cc8
0xa4 337 0x2638 SyncBlock 0000000019325cc8
0x15a 207 0x2668 SyncBlock 000000002bceecd8
0x161 156 0x266c SyncBlock 000000002bceecd8
0x1ae 63 0x2670 SyncBlock 000000002bceecd8
0xc2 356 0x2688 SyncBlock 0000000019325cc8
0x11c 50 0x26a8 SyncBlock 000000002bceecd8
0x1b8 242 0x26b0 SyncBlock 000000002bceecd8
As far as I can tell, none of our code is involved; I can provide the memory dump of our application as well as more detailed WinDbg output of my analysis, but those files are too large to attach here.