Oradirect.Net 2.40 Memory leaks

mabocho · Post by **mabocho** » Tue 10 Jun 2008 08:06

Hi,

We are having a big issue with system.outofmemory exception in Production environment.

After an analysis held by Microsoft (we have the report at the end) we definitely have a memory leak issue.

We are testing the new version of Corelab 4.70 for framework 1.1, and this raises another question for deploying because we have in the server Corelab 3.55 for framework 2.0 installed... Is there any problem on having these two versions in the same server installed?

Many thanks,
Miguel

Report:

I have had the dump file analysed and this is the result:
Out of the 690 MB in the largest memory dump close to 500 MB is .net memory. The rest is assemblies and other native data used for the garbage collector and other native memory.
Out of the 500 MB of .net objects, close to 475 MB are FREE objects, i.e. .net memory that is no longer used but can’t be discarded or compacted because it lies between pinned objects. In other words the memory is highly fragmented.

!dumpheap –stat

0x79ba2ee4 3,986 95,664 System.Collections.ArrayList

0x01b231e8 3,029 105,300 System.Int32[]

0x0386c4b4 3,351 107,232 System.Xml.Serialization.TempAssembly/TempMethod

0x029707c0 2,132 110,864 a.a.CoreLab1Oraclek

0x79bb408c 4,713 113,112 System.Reflection.RuntimePropertyInfo

0x79bd657c 4,746 113,904 System.Reflection.RuntimeFieldInfo

0x79bab93c 2,274 118,248 System.Collections.Hashtable

0x04e5f71c 2,131 127,860 CoreLab.Oracle.OracleConnection

0x79bbe59c 4,756 133,168 System.Reflection.Cache.ClearCacheHandler

0x01b25a50 4,756 133,168 System.Reflection.Cache.InternalCacheItem[]

0x01b25fd4 90 185,400 System.Decimal[]

0x79bd8524 4,409 211,632 System.Reflection.ParameterInfo

0x02976aac 5,604 313,824 CoreLab.Oracle.OracleType

0x79bb0d7c 14,154 339,696 System.Reflection.RuntimeMethodInfo

0x01b234a8 2,301 815,688 System.Collections.Hashtable/bucket[]

0x01b2209c 28,192 1,878,832 System.Object[]

0x02976dc4 58,312 2,565,728 CoreLab.Oracle.OracleAttribute

0x01b2292c 2,533 6,374,180 System.Byte[]

0x79b94638 137,403 9,518,780 System.String

0x000cc8d0 2,269 474,844,584 Free

Total 342,424 objects, Total size: 499,637,628

Fragmented blocks larger than 0.5MB:

Addr Size Followed by

0x107a690c 0.7MB large free object followed by 0x1084ed04 System.Byte[]

0x1084ed18 0.6MB large free object followed by 0x108f1b20 System.Byte[]

0x10a2ddfc 1.5MB large free object followed by 0x10baa010 System.Byte[]

0x10baae54 0.8MB large free object followed by 0x10c70d48 System.Byte[]

0x10cd51f4 0.6MB large free object followed by 0x10d64600 System.Byte[]

0x10d743b8 0.6MB large free object followed by 0x10e05768 System.Byte[]

0x11532848 0.5MB large free object followed by 0x115bc5e8 System.Byte[]

0x115e7dc8 0.8MB large free object followed by 0x116abd70 System.Byte[]

0x116c2d68 1.3MB large free object followed by 0x11817cc4 System.Byte[]

0x11891914 0.6MB large free object followed by 0x11932ea8 System.Byte[]

0x1199e808 0.7MB large free object followed by 0x11a58388 System.Byte[]

0x11c69ea4 1.0MB large free object followed by 0x11d66d74 System.Byte[]

0x11ddefdc 0.5MB large free object followed by 0x11e666bc System.Byte[]

0x1202618c 0.7MB large free object followed by 0x120e3f98 System.Byte[]

0x120e3fac 0.8MB large free object followed by 0x121b4394 System.Byte[]

0x1220deac 0.6MB large free object followed by 0x122a0f18 System.Byte[]

0x12321ec4 0.5MB large free object followed by 0x123abf9c System.Byte[]

0x12406124 0.5MB large free object followed by 0x1248d244 System.Byte[]

0x1248d258 0.6MB large free object followed by 0x1252a774 System.Byte[]

…

To get rid of this memory the pinned objects surrounding the free blocks need to be unpinned. Pinning is a mechanism by which you essentially tell an object that it can’t be moved by calling GCHandle.Alloc. This is done so that the object doesn’t move while you call native methods, like windows apis or other similar things. When the call is done the code is supposed to unpin the object so it can be moved and garbage collected again by calling handle.Free(), but for some reason this does not happen here.

The question then becomes,
1. Who is pinning these objects?
2. Why are they not being unpinned?

I ran a query for GCHandle.Alloc and found that the following methods were calling GCHandle.Alloc(byte[], GCHandleType.Pinned)

a.a.CoreLab1Oraclem.a(OracleConnection) : Void
CoreLab.Oracle.Oci.a(Byte[], Int32, Boolean) : String
CoreLab.Oracle.Oci.a(Char[], Boolean) : Byte[]
CoreLab.Oracle.OracleCommand.a(CommandBehavior, Int32, Boolean, Int32, Int32, Int32&) : OracleDataReader
CoreLab.Oracle.OracleDataReader.b() : Void
CoreLab.Oracle.OracleLoader.Open() : Void
CoreLab.Oracle.OracleLob.Read(Byte[], Int32, Int32) : Int32
CoreLab.Oracle.OracleLob.Write(Byte[], Int32, Int32) : Void

So it is likely that it was one of these methods that allocated and pinned the buffer[] and then something happened that caused it not to unpin it.
These methods come from the component CoreLab.Oracle which is part of OraDirect .NET, so you should follow up with CoreLab to determine how to best work around this situation. Perhaps there is some guidance on how to best use the OraDirect .NET drivers to avoid this type of behavior.

In addition I also found some nullreference exceptions on the finalizer

Exception 105ebff0 in MT 79bcabd4: System.NullReferenceException
_message: Object reference not set to an instance of an object.
_stackTrace:
05b77b3b [DEFAULT] [hasThis] Void CoreLab.Oracle.OracleConnection.a(I4)
01effc74
04e5f580
061d1f69 [DEFAULT] [hasThis] Void CoreLab.Oracle.OracleObjectBase.Finalize()
01effc78
04e5f990
_stackTraceString:
at CoreLab.Oracle.OracleConnection.a(Int32 A_0)
at CoreLab.Oracle.OracleObjectBase.Finalize()

When an exception happens on the finalizer, if it is not caught, you may run into a scenario where you get a memory leak.

Assume that you have the following destructor for MyClass for example

~MyClass(){
CallSomethingThatThrowsAnException();
ReleaseAllObjects();
}

If CallSomethingThatThrowsAnException() throws an exception, and you don’t have a try/catch around it, the destructor for MyClass will quit at the exception and thus never execute the call to ReleaseAllObjects, and therefore you may not release all the native objects, (including pinned handles) you should release. In other words you would be creating a memoryleak. Because of this it is extremely important that you don’t throw exceptions on the finalizer (as in this case) and that in the case when you do so, that you have try/catch blocks around them.

I would recommend that you also bring this up with the CoreLab support team to determine the best way to avoid throwing these exceptions, as they may be what is causing them not to free the GC Handles.

As a word of caution, if this exception would have occurred in 2.0 instead of 1.1. you would have been looking at a crash rather than a memory leak, as in 2.0, instead of silently quitting the finalizer, the process crashes because of the unhandled exception. This is actually a good thing as this is easier to troubleshoot than memory leaks caused by silent failures like this.

I hope this is clear. Please follow this up with CoreLab to get a reasoning behind this issue.

Regards,

Alexey.mdr · Post by **Alexey.mdr** » Wed 11 Jun 2008 07:48

Hello,

Please specify the full version of OraDirect .NET you are using.
All results available after invoking GCHandle.Alloc(, GCHandleType.Pinned) are stored in one place inside a cursor or a reader.
CoreLab.Oracle.OracleLoader has its own place of storage.
These resources are freed as the reader is closed.
It's likely that the memory size is made excessively large by some unclosed OracleDataReaders.
Do you use cursors that are returned as a procedure or script parameter?
Probably we have a problem here. Could you please send me (alexeyman*crlab*com) a small sample project to demonstrate the problem?
The CoreLab.Oracle.OracleObjectBase.Finalize() method in OraDirect .NET 3.55.26 catches all exceptions and nothing should be left.

Regards,
Alexey.

mabocho · Post by **mabocho** » Wed 11 Jun 2008 08:09

Hello,

The full version of Oradirect.Net we are using is 2.40.4.0 for framework 1.1 in this application.

The server has version 3.55.26 for framework 2 that is being used by other applications.

Ýesterday we have purchase version 4.70 for framework 1 and apply this new version for the application that has memory leaks.

My other question is: Can we have installed version 3.55 for .net2 and 4.70 for .net1.1 at the same time in the server?

I pass you an example by email.

Regards,
Miguel

Alexey.mdr · Post by **Alexey.mdr** » Wed 11 Jun 2008 12:29

Hello Miguel,

I have received the code example. We are investigating the problem.
Could please check whether the issue is reproducible with the latest version of OraDirect .NET?
There were crucial changes and many fixes in 3.** and 4.** versions.

Yesterday we have purchase version 4.70 for framework 1 and apply this new version for the application that has memory leaks.

Is the bug fixed in the new version?

Can we have installed version 3.55 for .net2 and 4.70 for .net1.1 at the same time in the server?

Yes, you can. It is recommended to remove any OraDirect .NET policy files from GAC.
Example of CoreLab policy file name in GAC policy.3.55.CoreLab.Oracle

Regards,
Alexey.

mabocho · Post by **mabocho** » Wed 11 Jun 2008 13:59

Hello Alexey,

We are waiting to receive the license and link to download the new version and then we'll let you know.

On the other hand, could you please explain me what to do to remove any OraDirect .Net files from GAC?

Thanks,
Miguel

Alexey.mdr · Post by **Alexey.mdr** » Wed 11 Jun 2008 14:09

Hello again,

Please follow these steps to remove Core Lab policy files:
- open Windows Explorer;
- navigate to %windows%assembly folder
- find policy files that has Core Lab in its name, like
policy.3.55.CoreLab.Oracle
- press Delete button on this file (or right mouse click -> Uninstall)

Regards,
Alexey.

mirkof · Post by **mirkof** » Fri 12 Sep 2008 13:33

Hello,

any news about this ? Because we do have the same problem (version 4.60)...

cheers
mirko

Shalex · Post by **Shalex** » Mon 15 Sep 2008 10:55

Hello,

did you follow the steps mentioned above? Are there policy files in your GAC?

Best regards,
Alexander

mirkof · Post by **mirkof** » Tue 16 Sep 2008 11:47

Yes

It was no problem to install the new version. But nevertheless the problems with the open byte arrays still exists in 4.60. My question was, if the original problem (open byte arrays) is solved....

cheers

mirko

Shalex · Post by **Shalex** » Tue 16 Sep 2008 12:37

Please give us full description of the problem you are experiencing:
1. What is the text of error?
2. What is the version of your OS, your Visual Studio, version and edition of OraDirect .NET?
3. Could you please send me (alexsh at devart.com) a test project to reproduce the error and a script that creates the tables your program affects?