ThreadLocal errors on Tomcat 6.0.24

rakesh's picture

I upgraded to 6.0.24 earlier today and started seeing the following errors in the logs on container shut down. Any ideas on whether this is something that is fixable, or even of concern?

12-Feb-2010 14:23:20 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [nextapp.echo.app.util.DomUtil$2] (value [nextapp.echo.app.util.DomUtil$2@10393e97]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@1e3c33d3]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
12-Feb-2010 14:23:20 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [nextapp.echo.app.util.DomUtil$3] (value [nextapp.echo.app.util.DomUtil$3@26e7127]) and a value of type [com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl] (value [com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl@20c6b8b0]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
12-Feb-2010 14:23:20 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [nextapp.echo.app.util.DomUtil$2] (value [nextapp.echo.app.util.DomUtil$2@10393e97]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@34baf4ae]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
12-Feb-2010 14:23:20 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [nextapp.echo.app.util.DomUtil$3] (value [nextapp.echo.app.util.DomUtil$3@26e7127]) and a value of type [com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl] (value [com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl@5e15e68d]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
12-Feb-2010 14:23:20 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [nextapp.echo.app.util.DomUtil$2] (value [nextapp.echo.app.util.DomUtil$2@10393e97]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@6fd3633c]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
12-Feb-2010 14:23:20 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [nextapp.echo.app.util.DomUtil$3] (value [nextapp.echo.app.util.DomUtil$3@26e7127]) and a value of type [com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl] (value [com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl@1f18317f]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.

ThreadLocal issues

First of all, ThreadLocal is used in DomUtil to prevent recreating DocumentBuilder and TransformerFactory objects all the time. This implementation improved Echo performance a lot (roughly 20%).

If you are only shutting down the container, there is no issue. If you are hot redeploying a lot however, this causes memory usage to increase. In extreme cases, an OOM may occur. The problem is that Echo does not release the ThreadLocal objects.

Echo should probably be modified to release these objects upon web application shutdown. The problem however is that there's no cleanup method on ThreadLocal. One solution is to simply null the ThreadLocal field and wait for the garbage collector to clean up. This would not get rid of the Tomcat warnings, but effectively fixes the memory leak.

The second solution is enumerating all active threads (race conditions may occur, see Thread Javadoc) and do some very nasty reflection based stuff (the same nasty stuff Tomcat performs to clean the ThreadLocal).

The third solution is to remove the use of ThreadLocal altogether and replace it with a simple cache mechanism (object pool). Unfortunately, Echo is still bound to Java 1.4, so we cannot use the concurrent collection utilities for this. This solution allows to perform a clean shutdown and hence this is (in my opinion) the preferred solution.

You might want to create a ticket for this in the tracker.

Niels

References:
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Thread.html#enumerate(java.lang.Thread[])
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/loader/WebappClassLoader.java?view=markup (method clearReferencesThreadLocals)

rakesh's picture

If you want to move to

If you want to move to concurrent utilities, I believe what was included in Java 1.5 was mainly an aggregation of third-party utilities that did the same for 1.4. There have been back ports as well (I had this bookmarked http://www.theserverside.com/news/thread.tss?thread_id=29059).

sgodden's picture

Indeed, the backport is

Indeed, the backport is here:

http://backport-jsr166.sourceforge.net/index.php

I don't want to introduce

I don't want to introduce concurrent stuff just for a few cache lookups. The time incurred for a bit of synchronization is probably negligible, although I haven't performed tests to verify this. Accessing ThreadLocal isn't free as well. Although we have a bit more administration to do when we implement a regular cache, objects must be returned to the pool since they'll no longer be tied to a thread. Another difficulty is coming up with a cache size / cache eviction strategy. I'm tempted to just keep creating objects as needed (just like we currently do), but that would never release objects after high load.

Thoughts?

rakesh's picture

How about setting the

How about setting the ThreadLocal's to null (which is what I have done when I used them)? Avoiding the leak is more critical than avoiding the warnings. I used to wonder why I was not able to redeploy my app more than 4-5 times without running out of memory. Echo is not the only culprit, since I saw similar errors for Lucene, Derby etc.

Will try

As mentioned before, that would be a reasonable workaround. I'll try to commit this workaround this week.

If you are getting errosr

If you are getting errosr with Tomcat than change it to Sun's Web Application Server, it supports upwards of 10 000 requests per day easily plus it works a bit faster, at least for me. Jena @ Model Business