loadUpdateData memory management for large CSV datasets
0 reputation · 09 Oct 2021, 23:37 UTC
The loadUpdateData change type in Liquibase allows for populating tables from CSV files by matching primary keys to determine whether to insert or update records. When dealing with datasets containing millions of rows, the process relies on the JVM memory allocation and the underlying JDBC driver's batching capabilities.
Because Liquibase processes these files sequentially without a native pagination or offset mechanism for the source CSV, there is a risk of table locking and heap exhaustion during the primary key lookup phase on massive tables.
- Does Liquibase provide a configuration to bound the number of records processed per transaction within a single
loadUpdateDatachange set? - What is the recommended approach to prevent
OutOfMemoryErrorwhen the CSV size exceeds the available JVM heap?