Speeding Up CSV Processing with PowerShell 7’s ForEach-Object -Parallel

Problem: Slow row‑by‑row CSV work

When you need to transform or enrich tens of thousands of CSV rows, a simple ForEach-Object loop can become a bottleneck. Each row is processed one after another, leaving idle CPU cores on a multi‑core machine.

Thesis: Use ForEach-Object -Parallel to run each row in its own runspace

PowerShell 7 introduced the -Parallel switch on ForEach-Object. It creates a separate PowerShell runspace for every pipeline item, allowing true concurrent execution without external modules. The result is a noticeable speed‑up when the work per row is CPU‑bound and the system has spare cores.

How it works

• Each input object is sent to a fresh runspace.
• The script block you provide runs in that runspace.
• Variables from the calling session are not automatically visible; you must expose them with $using: or pass them via -ArgumentList.
• By default, up to five runspaces run at once (-ThrottleLimit 5). Increase this value to match your core count, but be aware of memory and CPU pressure.
• Output order is not guaranteed; if order matters, collect results and sort afterward or use the experimental -PreserveOrder switch (available in preview builds).

Worked example: Calculating a column total

Suppose you have a CSV file data.csv with 10,000 rows and a numeric column named Value. You want to compute the sum of Value * 2 for each row.

# Sequential baseline (runs in the current session)
$seqResult = Import-Csv -Path data.csv | ForEach-Object {
    $_.Value * 2
} | Measure-Object -Sum

# Parallel version
$parallelResult = Import-Csv -Path data.csv | ForEach-Object -Parallel {
    # $using: is not needed here because we only use the current object
    $_.Value * 2
} -ThrottleLimit 8 | Measure-Object -Sum

Write-Host "Sequential sum: $($seqResult.Sum)"
Write-Host "Parallel   sum: $($parallelResult.Sum)"

Important points in the example:

• The -ThrottleLimit 8 argument tells PowerShell to keep up to eight runspaces alive, matching a typical 8‑core laptop.
• No external synchronization is needed because each iteration only reads its own object and returns a computed value.
• The final Measure-Object call runs after the parallel pipeline finishes, aggregating the results from all runspaces.

Trade‑offs and limitations

• Resource usage: Each runspace carries a baseline memory overhead. Raising -ThrottleLimit too high can cause swapping or CPU contention.
• Variable sharing: If you need to read a shared counter or configuration, you must pass it explicitly (-ArgumentList $config) or reference it with $using:config. Forgetting this leads to stale or null values.
• Ordering: The output sequence may differ from the input order. Sorting after the fact (| Sort-Object) restores predictability but adds a small overhead.
• Version requirement: -Parallel is available only in PowerShell 7.2+. Windows PowerShell 5.1 will ignore the switch and run sequentially, so scripts that rely on the speed‑up should check $PSVersionTable.PSVersion.

When to adopt it

Use ForEach-Object -Parallel when:

1. The per‑item work is independent and CPU‑intensive (e.g., calculations, API calls, compression).
2. You are running on a machine with multiple cores and sufficient RAM.
3. You can tolerate unordered output or are willing to sort afterward.
4. You are targeting PowerShell 7.2+ and can enforce that requirement in your environment.

If the work is mostly I/O‑bound (reading/writing files, database queries) or the dataset is tiny, the overhead of creating runspaces may outweigh any benefit, and a simple sequential loop remains preferable.

Actionable closing

Start small: pick a representative subset of your data, run the sequential and parallel versions side‑by‑side with Measure-Command, and compare both runtime and CPU usage (via Task Manager or Get-Process). Adjust -ThrottleLimit until you see diminishing returns or rising memory pressure. Once you have a sweet spot, roll the pattern out to the full dataset, remembering to handle any shared data with $using: or -ArgumentList and to re‑order results if needed.