Reading Thousands of Files in Parallel with F# Async Workflows: A Practical Guide

Problem: Sequential File Reads Are Too Slow

When an application needs to ingest dozens or hundreds of large log files, the naive approach is to read them one after another. On a 64‑bit machine with a 4‑core CPU and 16 GB of RAM, a single sequential read can take several minutes. Developers often ask: Can we keep the code simple while still getting the performance boost of parallelism?

Thesis: Async Workflows Let You Write Readable Parallel Code

F#’s async { … } workflow, introduced in F# 4.0, allows you to express asynchronous I/O in a style that looks almost synchronous. Under the hood, the compiler rewrites the workflow into a state machine that runs on the .NET thread pool. This gives you two key benefits:

• Readability – the code follows the natural flow of the task.
• Composable concurrency – you can combine many async computations with Async.Parallel or Async.Sequential.

1. Async Workflows Basics

At its core, an async workflow is a function that returns Async<'T>. The async { … } block can contain let! bindings that await other async computations, do! for side effects, and return to produce a value. Example:

let readLineAsync (path:string) : Async = async {
    use reader = System.IO.StreamReader(path)
    let! line = reader.ReadLineAsync() |> Async.AwaitTask
    return line
}

Notice Async.AwaitTask bridges the .NET Task API into the async world. This interoperability is essential when you consume third‑party libraries that expose Task<T>.

2. Parallel File Reads with Async.Parallel

The real power comes when you launch many async computations in parallel. Async.Parallel takes a sequence of Async<T> and returns a single Async<T[]> that completes when all children finish.

open System.Diagnostics
open System.IO

let readFileAsync (path:string) : Async = async {
    use stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, useAsync=true)
    let buffer = Array.zeroCreate<byte>(int stream.Length)
    let! bytesRead = stream.ReadAsync(buffer, 0, buffer.Length) |> Async.AwaitTask
    return buffer
}

let filePaths = [
    "C:\\data\\file1.log"
    "C:\\data\\file2.log"
    // … add more paths
]

let runParallelRead () = async {
    let sw = Stopwatch.StartNew()
    let! results = filePaths |> List.map readFileAsync |> Async.Parallel
    sw.Stop()
    printfn "Read %d files in %f seconds" results.Length sw.Elapsed.TotalSeconds
}

// Start the workflow on the thread pool
Async.Start(runParallelRead())

Running the above on a machine with a 4‑core CPU typically shows a 3‑x to 4‑x speedup compared to a sequential loop that calls File.ReadAllBytes for each file.

3. Controlling Concurrency: Throttling and Thread‑Pool Awareness

While Async.Parallel is convenient, launching dozens of thousands of async tasks can exhaust the thread pool and degrade performance. Two common strategies help:

• Async.StartChild – Wrap a long‑running async in Async.StartChild to limit how many run concurrently. The parent can await a limited number of children at a time.

  let limitedParallel (maxConcurrent:int) (tasks:Async<_> list) : Async<_[]> = async {
      let semaphore = new System.Threading.SemaphoreSlim(maxConcurrent)
      let enqueue t = async {
          do! semaphore.WaitAsync() |> Async.AwaitTask
          try return! t
          finally semaphore.Release() |> ignore
      }
      return! tasks |> List.map enqueue |> Async.Parallel
  }

• Task.Yield – Insert do! Async.Yield() inside loops to give the runtime a chance to schedule other work, preventing starvation of the main thread.

4. Exception Handling and Aggregation

Async workflows propagate exceptions deterministically. If any child in Async.Parallel throws, the entire computation fails and the exception is wrapped in an AggregateException. You can catch it with a standard try/with block:

try
    let! results = Async.Parallel tasks
    // process results
with
| :? System.AggregateException as ae ->
    printfn "One or more reads failed: %A" ae

For more fine‑grained control, use Async.Parallel with Async.Catch on each child to capture individual failures without aborting the entire batch.

Trade‑offs and Limitations

• Thread‑pool exhaustion – Uncontrolled parallelism can saturate the pool. Use throttling or limit the degree of parallelism.
• Blocking calls – Ensure you use the asynchronous file APIs (ReadAsync, WriteAsync) or wrap blocking calls in Async.StartAsTask to avoid blocking threads.
• Runtime support – The async/await syntax requires .NET 4.5+. On older runtimes the compiler will error.

Actionable Takeaway

When you need to read many files concurrently in F#, prefer async { … } with Async.Parallel and a simple throttling wrapper. Measure the elapsed time with Stopwatch, monitor thread‑pool usage, and always handle AggregateException to keep the application resilient.