Question
KV bulk import aborts with 'Error: KV bulk import failed: invalid line format at line 42'
Sora Indigo
0 reputation · 04 Mar 2023, 12:22 UTC
36.3K views1
Loading…
0 reputation · 04 Mar 2023, 12:22 UTC
When restoring a Cloudflare Workers KV namespace from a bulk export file, the wrangler kv:bulk import command stops processing as soon as it encounters a line that does not conform to the JSONL format, emitting an error such as Error: KV bulk import failed: invalid line format at line 42. This behavior can leave the target namespace with only a subset of the original keys, requiring manual cleanup or a full re‑import after correcting the source file.
The current workflow does not provide a built‑in mechanism to skip or isolate malformed lines, nor does it report all problematic lines in a single run. Administrators must therefore decide whether to pre‑validate the export file, to modify the import process to tolerate errors, or to accept the risk of partial restores and handle the aftermath manually.
What validation steps should be taken before importing a KV bulk export to guarantee line‑format correctness? Should the import command be adjusted to skip invalid lines and continue, or to collect and report all errors before aborting? How can a partially restored namespace be safely cleaned up without manual inspection of each key?
32775 reputation · 04 Mar 2023, 23:34 UTC
cloudflareworkersjq . < export.ndjson >/dev/null 2>&1 || echo "Invalid line at $(grep -n . export.ndjson | head -1)" (or loop over lines).sed -i '/^$/d' export.ndjson or filter with grep -v '^$'.grep -v '^$' export.ndjson | jq -c . > export.clean.ndjsonThe wrangler kv:bulk import command parses the export as JSONL and stops at the first line that fails JSON parsing, reporting “invalid line format at line X”. It has no flag to continue or collect multiple errors, so pre‑validation is required.
If you can delete and recreate the namespace, run:
wrangler kv:namespace delete # then recreate with wrangler kv:namespace create
Otherwise delete all keys (be aware of rate limits):
wrangler kv:key list --prefix='' | while read key; do
wrangler kv:key delete "$key"
done
If you lack permission to delete the namespace, please confirm so we can suggest an alternative approach.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 04 Mar 2023, 12:38 UTC
The wrangler kv:bulk import command expects every line to be a *single* JSON object with a key and value field. Any deviation—missing field, extra comma, or even a blank line—triggers the invalid line format error and stops the whole run. The CLI currently has no flag to skip bad lines.
jqInstead of manually hunting errors, pipe the file through jq to filter only syntactically correct lines:
jq -c . export.ndjson 2>&1 | tee log.txt | grep -v '^parse error' > clean.ndjson
When jq encounters a bad line it writes a parse error to stderr. The pipe above keeps a log of those errors, so you know exactly which lines failed without stopping the entire process.
After the run, inspect log.txt to see the line numbers and error messages. If you need to correct the source, you can edit only those few lines instead of re‑exporting the whole namespace.
When the import aborts, the namespace usually contains a subset of keys. A quick way to reset it is:
wrangler kv:bulk export --namespace my‑ns > current.ndjsonwc -l current.ndjsonwhile read k; do
wrangler kv:key delete "$k"
sleep 0.1 # throttle to stay under 50 req/s
done < <(jq -r .key current.ndjson)
wrangler kv:bulk import --namespace my‑ns clean.ndjsonFor very large namespaces, consider using a bulk‑delete file generated by wrangler kv:bulk export and a small helper script that deletes keys in batches.