Which profile_batch range avoids warmup overhead in tf.keras.callbacks.TensorBoard?

Measuring Training Bottlenecks

Identifying whether a Keras model is compute-bound or input-bound is essential before applying optimizations to the tf.data pipeline or model architecture. The tf.keras.callbacks.TensorBoard callback provides a profile_batch argument to capture a profiler trace during fit(), which can then be analyzed in the TensorBoard Profile tab to see a breakdown of step time.

Steady-State vs. Warmup Traces

Profiling early batches often captures one-time overheads, such as graph compilation, tf.data cache initialization, and initial GPU memory allocation. These factors can skew the results, making it difficult to determine the actual steady-state throughput of the training loop.

While a range (e.g., '50,60') can be specified to sample mid-run batches, there is no documented standard for selecting a range that reliably bypasses warmup effects across different hardware configurations and dataset sizes.

• Does the profiler trace include the overhead of the profiling mechanism itself?
• What is the recommended strategy for determining a representative profile_batch range to ensure measurements reflect steady-state performance?