Diagnosing and Fixing Metaspace/PermGen OutOfMemoryErrors from Classloader Leaks in Tomcat

Problem Statement

After several hot redeploys of a web application, Tomcat eventually throws
java.lang.OutOfMemoryError: Metaspace (Java 8+) or
java.lang.OutOfMemoryError: PermGen space (Java 7). The JVM can start fine, but the error appears after a handful of redeployments, and the application cannot be restarted until a full server reboot.

Root Cause

Classloader leaks. When a webapp is undeployed, its classloader should be eligible for garbage collection. If a static reference or a system‑level thread retains a reference to the webapp’s classloader, the JVM keeps it alive. Repeated redeploys create new classloaders, exhausting the Metaspace/PermGen.

Diagnostic Flow

Use the following ordered checks to confirm a classloader leak and identify the offending component.

1. Verify Memory Growth

1. In a terminal, run:

   jcmd  GC.class_histogram | grep WebappClassLoader

2. Record the number of WebappClassLoader instances before and after a redeploy.
3. Repeat for 5–10 redeploys. A steady increase indicates a leak.

2. Inspect Thread Dumps

1. Take a thread dump after the error:

   jstack -l  > dump.txt

2. Search for threads whose names contain the application name or custom prefixes (e.g., "myapp-worker").
3. Check if those threads are still alive after the context was destroyed.
4. Look for Thread.State: RUNNABLE or BLOCKED states that persist beyond the undeploy.

3. Detect System‑Level Classloader Registrations

1. Use jcmd <pid> VM.system_properties to check for java.util.logging.manager or org.apache.logging.log4j.core.LoggerContext pointing to a webapp class.
2. Search the classpath for JDBC drivers or logging frameworks that are loaded both in WEB-INF/lib and TOMCAT_HOME/lib.
3. If duplicates exist, the driver may register itself with the system classloader, keeping a reference to the webapp classloader.

4. Use VisualVM or JConsole

1. Connect to the JVM and open the Memory tab.
2. Observe Non‑Heap Memory Usage over successive redeploys.
3. In the Classes tab, filter by the application’s package name. A growing count signals a leak.

Fixes Tied to Findings

Fix A – Graceful Thread Shutdown

Ensure every custom thread or ExecutorService is terminated in a ServletContextListener#contextDestroyed implementation.

public class AppShutdownListener implements ServletContextListener {
    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        ExecutorService pool = (ExecutorService) sce.getServletContext()
            .getAttribute("myPool");
        if (pool != null) {
            pool.shutdown();
            try { pool.awaitTermination(30, TimeUnit.SECONDS); } catch (InterruptedException e) {}
        }
    }
}

Risk: Failing to await termination may leave threads alive; however, this is much safer than a blanket JVM restart.

Fix B – Centralize Shared Libraries

Move JDBC drivers, logging frameworks, and other shared JARs from WEB-INF/lib to TOMCAT_HOME/lib. This prevents duplicate classloaders and removes static references that tie the webapp classloader to the system classloader.

After moving, redeploy the app and confirm the class_histogram no longer increases.

Fix C – Avoid Static Class References

Review the code for static fields that hold references to classes loaded by the webapp’s classloader (e.g., static final Logger = LoggerFactory.getLogger(...)). Replace them with instance fields or use the ThreadLocal pattern if necessary.

Fix D – Use a ServletContextListener for Resource Cleanup

Besides threads, ensure that database connection pools, scheduled tasks, and any Timer objects are closed in contextDestroyed.

Fix E – JVM Heap Tuning (Temporary Work‑Around)

Increase the Metaspace limit to postpone the crash:

java -XX:MaxMetaspaceSize=512m -jar myapp.war

This does not solve the leak and should be used only while a permanent fix is being implemented.

Verification After Fixes

1. Redeploy the application 5–10 times.
2. Run jcmd <pid> GC.class_histogram after each redeploy and confirm the number of WebappClassLoader instances remains constant.
3. Monitor Non‑Heap Memory Usage in VisualVM; it should stay stable.
4. Check thread dumps for the absence of lingering application‑named threads.

Escalation Criteria

• If the number of classloaders continues to grow after applying all fixes, the leak likely originates from a third‑party library that cannot be modified.
• In that case, isolate the library in a separate webapp and use a custom ClassLoader to unload it, or replace the library.
• Consider upgrading to the latest Tomcat version, which includes improvements to classloader cleanup.

Limitations and Caveats

• Increasing -XX:MaxMetaspaceSize is only a stop‑gap and may mask the underlying problem.
• Force‑restarting the JVM clears the leak but does not fix the code that keeps references alive.
• Some classloaders (e.g., those created by frameworks like Spring Boot’s SpringApplication) may not be directly visible in the histogram; in such cases, use jmap -histo:live for a deeper inspection.