Graph API Batch Requests vs. Individual Calls for Latency under Concurrent Load

Goal

We want to retrieve user profile data for thousands of users in a single operation. The operation must finish in under 1 second when the service receives 200 concurrent requests.

Context

Under heavy load, the average round‑trip time for a single Graph API call spikes to 300 ms, causing the overall pipeline to exceed the SLA. Two documented approaches exist: sending each user’s request individually or grouping them into a single batch request (up to 50 calls per batch).

Constraints & Uncertainty

• Batch size is limited by the Graph API (max 50 calls per batch).
• Each batch still incurs a single HTTP round‑trip, but the server must process all sub‑requests.
• Error handling differs: a failure in one sub‑request does not abort the entire batch.
• The service must preserve the order of responses for downstream processing.

Specific Questions

1. At what concurrency threshold does batching start to outperform individual calls in terms of total latency?
2. How does varying the batch size (e.g., 10 vs. 50 sub‑requests) affect the average round‑trip time under 200 concurrent connections?
3. What trade‑offs exist between batch overhead (JSON serialization, server parsing) and the reduced number of HTTP handshakes?