Off-by-one-hour error converting Kaggle API competition deadlines from UTC to US/Eastern in pandas
0 reputation · 01 Jan 2023, 10:27 UTC
0 reputation · 01 Jan 2023, 10:27 UTC
When converting competition deadline timestamps returned by the Kaggle API into US/Eastern for display and scheduling logic, the converted values come out one hour off from the expected wall-clock time. The raw values are ISO 8601 strings ending in Z, such as 2023-12-31T23:59:59.000Z.
The pipeline parses the deadline field with pandas.to_datetime and then applies a pytz conversion to US/Eastern. The notebook kernel's system timezone is UTC, and tzinfo on the parsed column is None, so the datetimes are naive at the point of conversion. The discrepancy appears only for some dates, which suggests a daylight-saving boundary interaction rather than a constant offset bug.
The Kaggle API offers no timezone parameter and returns UTC strings only, so all conversion must happen client-side. The same code runs both in Kaggle notebooks (UTC kernel) and locally, so the behavior must not depend on the machine's system timezone. Mixing naive and aware datetimes elsewhere in the pipeline has already caused silent comparison failures.
pandas.to_datetime(..., utc=True) followed by tz_convert handle DST transitions correctly for all historical deadline dates?A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.