How to prevent memory leaks when reloading web applications
Posted by cornel | Filed under Java
During my career as a software engineer I had to fix some nasty memory leaks that appeared only when trying to reload the current web application (one bug is described here). Most of the time the PermGen space was completely filled with class definition and the following error message was returned: “java.lang.OutOfMemoryError: PermGen space” (on older JVM it is possible to receive just “java.lang.OutOfMemoryError”)
PermGen is a region of memory allocated by the JVM for class definitions and for the constant pool. At first glance you might will be tempted to increase its size (the default value is 64MB) using the -XX:MaxPermSize parameter, however this solution will only allow you maybe just to reload your application perhaps several times more. The better solution is to identify the memory leak (and this post will describe several possible causes) and after that to fix it .
I will also write about the classloader architecture of an application server and what happens when the application server tries to unload a deployed application but I will not go into much detail, since there are better articles or books describing these things.
a)Classloader arhitecture for an applications server.
Classloaders are organized in parent child relationship, and a class can be visible only if it is visible from the current classloader or any of its parents – it cannot go down in hierarchy. Every classloader keeps a list of references to the loaded classes (take a look in java.lang.ClassLoader).
The following diagram (inspired from Tomcat documentation) describes the simplified classloader hierarchy used by an application server :
Bootstrap
|
System
|
Common
/ \
Webapp1 Webapp2 ...
If you are curious to see the whole hierarchy of classloaders for your class and server you can use the following code:
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
while(classLoader != null) {
System.out.println(classLoader);
classLoader = classLoader.getParent();
}
The application server uses this hierarchy of classloaders in order to keep the applications isolated. For example if a class is loaded by the classloader Webapp1 it will be not visible from a class loaded by the classloader Webapp2. However a class loaded by the classloader Common is visible from the classloader Webapp2. As an analogy all the libraries from the Tomcat lib folder are visible for all deployed applications, but one application cannot see the libraries from the lib folder located inside another application.
b)Deploying and restarting your application
When you deploy and start your own application the server will a create a new classloader – let’s call it WEBCLOADER1, which will be used to load the classes from your application. When you redeploy your application it will set the reference of WEBCLOADER1 to null and it will create a new classloader used again to load your application. When you undeploy your application it will destroy WEBCLOADER1 and it will try to delete the application.
This process works well if the classes from your application are referenced only from WEBCLOADER1 or any other children classloaders created from WEBCLOADER1. If not, for example if one of your classes is referenced from the System classloader then your deployed application cannot be garbage collected and eventually you will receive the dreaded OutOfMemoryError.
How is it possible to have this situation? Is too simple unfortunately, just use something like this:
private static final ThreadLocal cache = new ThreadLocal();
cache.set(someObject);
In this case the current thread (one of the threads created by the application servers) has a reference to your object. The thread was created by the application server using a higher level classloader so it does not matter that WEBCLOADER1 is null, your application cannot be garbage collected.
The problem also arises when you use methods from the java.beans.Introspector class without calling flushCaches because it uses a static cache in order to make the introspection faster, so again your classes are going to be referenced from a classloader other than WEBCLOADER1.
c)How do I identify a memory leak related to the multiple classloader problem?
Use a good profiler. I was using YourKit profiler, it was one of the few able to help me two years ago (today there maybe be other choices). Using YourKit you can take a memory snapshot, undeploy your application, take another snapshot and use the compare snapshots features. If you will see objects belonging to your application it means that you have memory leaks. You can use the “Find paths from GC roots” option to see who’s keeping references to your class. That’s the easier part, after that you will have to locate the problem in your application – or in other third-party libraries used by your application. Bellow I will wrote some tips related to common problems.
Use a ServletContextListener to intercept when the context is destroyed; this is the place where you should have your clean up code. Or you can use the destroy() method from a dedicated servlet. Bellow are some tips for some common problems:
- if you use commons-logging then call this method from the clean up method
LogFactory.release(Thread.currentThread().getContextClassLoader());
DriverManager.deregisterDriver(yourDriverClass);
February 17th, 2009 at 2:29 pm
I have a simple flex – java app. I am using tomcat 6. On the serverside I open a connection to a db, read some data and return them to the flex app. Every time I reload the app from tomcat, I see an increased memory size. (out of memory occurs eventually). I deregister the db driver, none of my classes are present in the comparison yourkit list. Thank you! (Great article!)
February 17th, 2009 at 3:16 pm
Thanks for the comments
Btw, if you are able to send me some source code in order to reproduce the problem I would be curious to take a look.