model.fit lacks native timeout or cancellation in Keras 2.x
0 reputation · 15 Oct 2024, 03:09 UTC
The goal is to define a supported way to cancel or enforce a wall-clock timeout for model.fit in Keras 2.x during long training runs without relying on external process control.
The fit loop is documented with a callback interface including on_epoch_begin and on_batch_end, and with callbacks such as EarlyStopping and TerminateOnNaN. No public timeout or cancellation token parameter is documented for model.compile or model.fit. The design of a built-in mechanism remains unresolved, including what semantics Keras should guarantee for interruption, resource cleanup and checkpointing.
Should Keras expose a native timeout or cancellation parameter for model.fit, and if so what cleanup guarantees should be required? Is the cooperative stop_training flag sufficient for hard cancellation across custom layers and distributed strategies?