Keras EarlyStopping and ModelCheckpoint: Keeping the Best Epoch Without Guessing

When you train a neural network in Keras, validation loss often improves for a while and then gets worse. The model you hold after training finishes is rarely the best one you saw. If you have ever stared at a training curve and thought, the best epoch was somewhere in the middle, the standard fix is to stop training when the metric stalls and to persist the best weights before they drift away.

EarlyStopping stops the loop, but does it restore the model?

EarlyStopping watches a metric and halts fit() when that metric stops improving. Three arguments matter most:

• monitor: the metric name, usually 'val_loss' or 'val_accuracy'.
• mode: the direction of improvement. Loss metrics need 'min'; accuracy or AUC need 'max'. If you omit mode, Keras infers it from the metric name, but explicit is safer.
• patience: how many epochs to wait after the last improvement before stopping.

Setting restore_best_weights=True rolls the in‑memory model back to the best epoch. That is the simplest fix when the deliverable is the final model object in the same Python process. It is not a full training resume: optimizer moments such as Adam accumulators are not restored. If you need to continue training later, treat this as an inference checkpoint, not a pause button.

ModelCheckpoint persists the best epoch to disk

ModelCheckpoint writes files. With save_best_only=True, it only writes when the monitored metric improves, so the file on disk reflects the best epoch rather than the last one. The monitor and mode should match your EarlyStopping configuration exactly.

A common trade‑off is save_weights_only=True. It produces smaller artifacts, but the exact architecture must be rebuilt in code before you call load_weights. If you save the full model, the file is larger and self‑contained, but you must load it with a compatible Keras version.

Option	Artifact	Rebuild needed?	Best for
save_weights_only=False	Full model (.keras or .h5)	No	Sharing or serving
save_weights_only=True	Weights only	Yes	Smaller storage, same codebase

Version sensitivity: Keras 2, Keras 3, and import paths

Serialization is not stable across major versions. Keras 3 moved the default format toward .keras, and .h5 artifacts are not interchangeable with .keras files. The import path also matters: tf.keras and standalone keras can place callbacks in different modules and may serialize differently. Pin the major version in your environment lock file, and verify artifact loading in a fresh process after any upgrade.

A worked example

The following script builds a tiny deterministic classifier on synthetic data. Run it in your training environment with Keras 3.x installed. It assumes numpy and keras are available; adjust the import if you use tf.keras.

# Run in your training environment (Python 3.9+)
import numpy as np
import keras
from keras import layers
from keras.callbacks import EarlyStopping, ModelCheckpoint

# Record the installed major version; callbacks and serialization can shift
print("Keras version:", keras.__version__)

rng = np.random.RandomState(42)
X = rng.randn(1000, 20).astype("float32")
y = (X[:, 0] + 0.5 * X[:, 1] > 0).astype("float32")

model = keras.Sequential([
    layers.Input(shape=(20,)),
    layers.Dense(8, activation="relu"),
    layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

callbacks = [
    EarlyStopping(
        monitor="val_loss",
        patience=5,
        restore_best_weights=True,
        mode="min",
    ),
    ModelCheckpoint(
        filepath="best.keras",
        monitor="val_loss",
        save_best_only=True,
        mode="min",
    ),
]

history = model.fit(
    X,
    y,
    epochs=100,
    batch_size=32,
    validation_split=0.2,
    callbacks=callbacks,
    verbose=2,
)

# Practical check: confirm the file exists and the loop stopped early
import os
print("Checkpoint exists:", os.path.exists("best.keras"))
print("Epochs run:", len(history.history["loss"]))

If validation loss stops improving for five epochs, training halts and best.keras remains on disk. Because restore_best_weights=True is set, the in‑memory model also reflects the best epoch. To verify in your environment, confirm that len(history.history['loss']) is smaller than the epochs argument, that best.keras exists, and that the minimum value in history.history['val_loss'] aligns with the epoch that produced the checkpoint. Do not assume a specific epoch count; run it once and inspect the history object.

One subtle trap: validation_split

validation_split=0.2 holds out the last 20% of the arrays before shuffling. If your rows are ordered by time, class, or any meaningful sequence, shuffle explicitly before calling fit(), or pass an explicit validation_data tuple. Otherwise your validation estimate is biased by row order.

Trade-offs and limitations

ModelCheckpoint adds disk I/O at epoch boundaries. For very large models, that write can be non‑trivial. EarlyStopping with restore_best_weights=True does not restore optimizer state, so it is not a substitute for a resumable checkpoint. And validation metrics are estimates on the chosen split, not guarantees of generalization.

Actionable checklist

1. Decide whether you need an in‑memory model, a file on disk, or both.
2. Set monitor and mode explicitly; do not rely on inference for custom metrics.
3. If using save_weights_only=True, keep the architecture code under version control so the weights can be rebuilt.
4. Pin the Keras major version and test artifact loading in a clean process.
5. If row order matters, shuffle before validation_split or supply validation_data.
6. Run once with save_best_only=False and once with True to confirm that only the best epoch file remains and that monitor and mode agree.

Pairing these two callbacks removes the guesswork about which epoch to keep. The remaining work is matching the configuration to your metric direction and your serialization format.