Diagnosing Keras ModelCheckpoint & EarlyStopping Failures: A Practical Checklist

Common Symptom

During training you notice that no checkpoint files appear in the target directory, or that the model stops after the first epoch even though validation loss is still improving. The training log shows no error messages related to the callbacks.

Root‑Cause Table

Symptom	Possible Cause	Diagnostic Check	Immediate Fix
No checkpoint files created	Invalid filepath (illegal characters, non‑existent directory, or insufficient permissions)	Run os.path.isdir() and os.access() on the directory; inspect the callback’s filepath attribute	Correct the path, create the directory, or adjust permissions
EarlyStopping halts after first epoch	Patience set to 0 or monitor metric never recorded	Print callback.monitor and model.metrics_names during training	Increase patience or use a metric that is logged
Callbacks trigger only on primary replica in multi‑GPU setup	Distribution strategy suppresses callbacks on secondary replicas	Check tf.distribute.get_replica_context().replica_id_in_sync_group inside callback logic	Wrap callbacks in tf.keras.callbacks.experimental.DistributedCallback or use tf.keras.callbacks.CallbackList with strategy
Monitor name misspelled or mismatched	Callback monitors a metric that the model never logs (e.g., val_accuracy when validation data is absent)	Print model.metrics_names after compilation	Correct the monitor string or add validation data
Checkpoint files not saved in TF 2.13+ with save_weights_only	Behavior changed between TF 2.10 and 2.13	Check tf.__version__ and experiment with save_weights_only flag	Explicitly set save_weights_only=False if full model is needed

Step‑by‑Step Diagnostic Checklist

1. Verify the training loop logs the metric you intend to monitor.

   model.compile(optimizer='adam', loss='mse', metrics=['mae'])
   print('Metrics:', model.metrics_names)
   

2. Inspect the callback’s attributes before training starts.

   checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath='ckpt/model.h5', monitor='val_loss')
   print('Checkpoint monitor:', checkpoint.monitor)
   print('Checkpoint filepath:', checkpoint.filepath)
   

3. Check file‑system access.

   import os
   path = os.path.dirname('ckpt/model.h5')
   print('Dir exists:', os.path.isdir(path))
   print('Writable:', os.access(path, os.W_OK))
   

4. Run a minimal training script with a known improving metric.

   import tensorflow as tf
   import numpy as np
   
   x = np.random.rand(100, 10)
   y = np.random.rand(100, 1)
   
   model = tf.keras.Sequential([tf.keras.layers.Dense(1)])
   model.compile(optimizer='adam', loss='mse', metrics=['mae'])
   
   checkpoint = tf.keras.callbacks.ModelCheckpoint('ckpt/model.h5', monitor='mae', save_best_only=True)
   early = tf.keras.callbacks.EarlyStopping(monitor='mae', patience=2)
   
   model.fit(x, y, epochs=10, callbacks=[checkpoint, early])
   

5. Observe training logs for callback messages.
   • ModelCheckpoint: "Saved best model to ckpt/model.h5"
   • EarlyStopping: "Epoch 3: early stopping (mae improved by 0.00)"
6. Validate the checkpoint file exists.

   import os
   print('Checkpoint file exists:', os.path.isfile('ckpt/model.h5'))
   

7. Confirm the callback is active on all replicas (if using tf.distribute).

   strategy = tf.distribute.MirroredStrategy()
   with strategy.scope():
       model = tf.keras.Sequential([...])
       # callbacks defined outside strategy scope
       model.fit(..., callbacks=[checkpoint, early])
   

Fixes & Best Practices

• Use absolute paths or ensure relative paths resolve correctly. Relative paths can change depending on the working directory when launching training jobs (e.g., via Kubeflow).
• Explicitly specify save_best_only when you want to keep only the best model. This avoids clutter and makes it clear that the file should be overwritten when a better metric is observed.
• When using custom metrics, pass the metric instance to the callback. For example:

  mae_metric = tf.keras.metrics.Mean(name='mae')
  model.compile(..., metrics=[mae_metric])
  checkpoint = tf.keras.callbacks.ModelCheckpoint(..., monitor='mae')
  

• Set restore_best_weights=True only if you have save_best_only=True; otherwise the model may revert to a non‑optimal state.
• In distributed training, register callbacks with tf.keras.callbacks.experimental.DistributedCallback or wrap them in a tf.keras.callbacks.CallbackList that applies to all replicas.

Escalation & When to Ask for Help

• If after correcting all checklist items the callbacks still fail, check the TensorFlow GitHub issues for any version‑specific bugs.
• On managed platforms (e.g., GCP AI Platform, AWS SageMaker), verify that the training container has the correct IAM or role permissions to write to the specified bucket or file system.
• When callbacks work locally but not in production, compare the TensorFlow and Keras versions; subtle API changes can alter default monitor names.

Practical Verification

After applying fixes, run the minimal script above again. If you see the checkpoint file created and the early‑stop message appears only after the defined patience threshold, the issue is resolved. If not, revisit the diagnostic table and ensure you’re not missing a subtle mismatch between monitor and the actual metric name.