Handling Million-Row Data Loads with Salesforce Bulk API v2

The Governor Limit Wall

When moving a few hundred records into Salesforce, the standard REST or SOAP APIs are sufficient. However, once you hit the million-row mark, these synchronous APIs become a liability. You will likely encounter REQUEST_LIMIT_EXCEEDED errors or trigger Apex CPU timeouts because the platform cannot process massive payloads in a single transaction without impacting other users.

The solution is the Bulk API v2. Unlike standard APIs that process records immediately and hold the connection open, Bulk API v2 uses an asynchronous model. You upload a large dataset, Salesforce chunks it into manageable batches in the background, and you check back later for the results. This decouples the data transfer from the data processing, allowing you to move millions of records without hitting the strict limits of interactive API calls.

How the Asynchronous Job Model Works

Bulk API v2 simplifies large-scale engineering by removing the need for the developer to manually calculate batch sizes. In v1, you had to specify exactly how many records went into each batch. In v2, you simply provide the data, and Salesforce handles the partitioning.

The lifecycle of a bulk operation follows a specific sequence:

• Job Creation: You define the operation (Insert, Update, Upsert, or Delete) and the target object. This returns a unique Job ID.
• Data Upload: You upload your data as a CSV file. For very large sets, compressing the file into a .zip format is recommended to reduce network latency.
• Background Processing: Salesforce queues the job. It processes the records in the background, respecting governor limits by spreading the load across the system.
• Result Retrieval: Once the job status moves to JobComplete, you request the successful and failed record sets.

Worked Example: Upserting 1 Million Accounts

To perform a large-scale upsert, you will interact with the REST endpoints. Ensure your integration user has the "API Enabled" and "Modify All Data" (or object-specific edit) permissions.

Step 1: Create the Job

Run this POST request to initialize the process. Replace External_ID__c with your actual unique identifier field.

POST /services/data/v60.0/jobs/ingest
Content-Type: application/json

{
  "object": "Account",
  "contentType": "CSV",
  "operation": "upsert",
  "externalIdFieldName": "External_ID__c"
}

Step 2: Upload the Data

Use the Job ID returned from the previous step. The body of this request is the raw CSV content.

PUT /services/data/v60.0/jobs/ingest/{jobId}/batches
Content-Type: text/csv

External_ID__c,Name,Phone
ACC-001,"Acme Corp",555-0101
ACC-002,"Globex",555-0102
... [remaining 999,998 rows] ...

Step 3: Monitor and Verify

Because the process is asynchronous, you must poll the job status. Run this GET request periodically:

GET /services/data/v60.0/jobs/ingest/{jobId}

Expected Check: Look for "state": "JobComplete". If the state is Failed or Aborted, the entire job has stopped. If it is JobComplete, some individual records may still have failed.

Trade-offs and Constraints

While powerful, Bulk API v2 is not a "silver bullet" for every scenario. The primary trade-off is latency. Because the job is queued, you cannot use this for real-time synchronization where a user expects an immediate confirmation on a screen.

Additional limitations include:

• File Size: The maximum size for a single upload is 10 GB. If your dataset exceeds this, you must split the data into multiple jobs.
• Daily Quotas: While Bulk API v2 reduces the number of API calls, it still counts toward your organization's total daily API request limit.
• Processing Order: Records are processed in parallel. If your data has strict dependencies (e.g., you must create the Account before the Contact), you cannot do this in a single Bulk job; you must wait for the Account job to complete before starting the Contact job.

Verifying the Result

Never assume a JobComplete status means 100% of records were processed. To verify the outcome, request the successful and failed record sets separately:

• Successful Records: GET /services/data/v60.0/jobs/ingest/{jobId}/successfulResults
• Failed Records: GET /services/data/v60.0/jobs/ingest/{jobId}/failedResults

The failedResults endpoint provides a CSV containing the original row data along with the specific error message (e.g., REQUIRED_FIELD_MISSING), allowing you to clean the data and re-run only the failed subset.