Limits on backing up the webapps directory while Tomcat is running

Answer to the Core Question

Copying the webapps directory while Tomcat is actively serving requests is not guaranteed safe. In‑flight JSP compilations or class‑loader writes can leave partially written .class files in the snapshot, producing a deployment that fails to start or throws ClassNotFoundException or InvalidClassFormat errors after restore.

Why Live Copy Is Risky

• JSP compilation: The first request to a JSP triggers a compiler that writes a .class file to the work directory or the exploded WAR. If the copy starts mid‑write, the resultant class file may be truncated.
• Exploded WAR changes: Tomcat may extract nested WARs, generate temporary files, or update the directory structure during startup or normal operation. A live copy can capture an intermediate state that isn’t deployable.
• File‑system consistency: Even a modern snapshot (LVM, ZFS, cloud block snapshot) is atomic only at the block level; it still captures whatever bytes were in flight at snapshot time, so a partially written file can survive the snapshot.

Verified Mitigation Strategies

1. Use a block‑level snapshot first, then copy
   • Take an LVM or ZFS snapshot of the partition containing CATALINA_BASE immediately before initiating the copy.

     # Example with LVM
     lvcreate --size 10G --snapshot --name webapps_snap /dev/vg1/catalina_base
     # Mount the snapshot and copy
     mount /dev/vg1/webapps_snap /mnt/snap
     rsync -a /mnt/snap/webapps /backup/webapps-$(date +%F)
     # After copy, delete the snapshot
     lvremove /dev/vg1/webapps_snap
     

   • Verify the snapshot size is stable (no large file growth) and check Tomcat logs for any "Compiled JSP" entries during the snapshot window.
2. Quiesce the specific webapp via the Manager API
   • Undeploy the target webapp (or set reloadable="false" if using a single instance) to stop all JSP compilation threads.

     # Using the Manager REST endpoint
     curl -u admin:password "http://localhost:8080/manager/text/undeploy?path=/myapp"
     # Wait a few seconds for cleanup threads to finish
     

   • Copy the exploded directory.

     rsync -a /opt/tomcat/webapps/myapp /backup/myapp-$(date +%F)
     

   • Redeploy the webapp.

     curl -u admin:password "http://localhost:8080/manager/text/deploy?path=/myapp&update=true&war=file:/opt/tomcat/webapps/myapp.war"
     

3. Full Tomcat stop (simplest but downtime)
   • Shut down Tomcat, copy CATALINA_BASE or webapps, then restart.

     systemctl stop tomcat
     rsync -a /opt/tomcat /backup/tomcat-$(date +%F)
     systemctl start tomcat
     

When to Prefer Which Method

• Production environments where any downtime is unacceptable: use a snapshot + copy, then test the backup in a staging instance before restoring to production.
• When the webapp is small and downtime of a few minutes is tolerable: undeploy, copy, redeploy.
• When the entire Tomcat instance can be stopped safely (e.g., during a scheduled maintenance window): stop, copy, start.

Verification Checklist

• After restoring the backup, deploy the webapp in a test environment and confirm it starts without class‑loading errors.
• Run sha256sum -r on the original and copied directories; mismatches alone don’t prove corruption but indicate changes during the copy.
• Review Tomcat logs for "Compiled JSP" entries that overlap the backup window; if found, the snapshot likely captured a partially written class.

Missing Diagnostic Detail

To fine‑tune the recommendation, could you confirm whether the webapp is deployed as an exploded directory or as a WAR file, and whether JSP compilation is enabled in your environment? This affects whether the undeploy‑copy‑deploy cycle will suffice or a snapshot is required.