Problems when writing parquet with timestamps prior to 1900 in AWS Glue 3.0
When switching from Glue 2.0 to 3.0, which means also switching from Spark 2.4 to 3.1.1, my jobs start to fail when processing timestamps prior to 1900 with this error: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob. You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps