Optimizing Elasticsearch Throughput with the Bulk API

The Ingestion Bottleneck

When moving large datasets into Elasticsearch, sending individual index requests creates massive overhead due to repeated HTTP handshakes and per-request processing. The Bulk API solves this by grouping multiple operations into a single request, typically increasing throughput by 10x to 100x. However, improper batch sizing can lead to JVM heap exhaustion or silent data loss through partial failures.

Bulk API Mechanics

The Bulk API uses a specific newline-delimited JSON (ndjson) format. Each operation requires two lines: an action metadata line and the document source line. Both must end with a newline character (\n).

POST /_bulk
{ "index" : { "_index" : "logs", "_id" : "1" } }
{ "timestamp": "2026-10-14T04:00:00", "level": "INFO", "message": "System start" }
{ "index" : { "_index" : "logs", "_id" : "2" } }
{ "timestamp": "2026-10-14T04:00:01", "level": "ERROR", "message": "Connection timeout" }

Technical Note: The action line defines the operation (index, create, update, or delete). If you omit the _id, Elasticsearch will auto-generate one, which is generally faster for high-volume ingestion.

Production Configuration Strategy

To maximize ingestion speed without crashing the cluster, apply these configuration patterns:

1. Batch Sizing and Concurrency

Avoid sending massive payloads. Aim for batches of 5,000 to 10,000 documents. The ideal size depends on your average document size; the goal is to keep the total request size between 5MB and 15MB.

2. Index Tuning for Loads

During initial bulk loads, disable the automatic refresh interval. This prevents Elasticsearch from creating new Lucene segments every second, which consumes heavy I/O.

# Run via Dev Tools or curl to disable refresh
PUT /logs/_settings
{
  "index": {
    "refresh_interval": "-1"
  }
}

3. Client-Side Implementation Example

When using a client library, you must explicitly check the response body. A 200 OK HTTP status only means the request was processed, not that every document was indexed.

# Python implementation using elasticsearch-py
from elasticsearch import Elasticsearch

es = Elasticsearch(['http://localhost:9200'])

def safe_bulk_index(docs, index_name):
    # Prepare actions in ndjson format
    actions = []
    for doc in docs:
        actions.append({"_index": index_name, "_source": doc})
    
    # Execute bulk request
    # Note: Use a timeout to prevent hanging on slow shards
    response = es.bulk(body=actions, request_timeout=60)
    
    # Critical: Check for partial failures
    if response['errors']:
        for item in response['items']:
            # The key is the action used (e.g., 'index')
            op_type = list(item.keys())[0]
            if item[op_type]['status'] >= 400:
                print(f"Error in doc: {item[op_type]['error']}")

# Example usage
my_docs = [{"msg": f"log {i}"} for i in range(5000)]
safe_bulk_index(my_docs, "logs")

Operational Limits and Common Pitfalls

JVM Heap Exhaustion

Bulk requests are buffered in memory. If your batch size is too large or you have too many concurrent bulk threads, you will trigger OutOfMemoryError. Monitor your heap usage using the nodes stats API:

GET /_nodes/stats/jvm

The "Silent" Partial Failure

The most common mistake is ignoring the errors boolean in the response. If one document in a batch of 10,000 has a mapping conflict (e.g., sending a string to a long field), Elasticsearch will reject that document but index the other 9,999. Without checking the items array, you will lose data without an alert.

Version Conflicts

If multiple bulk processes update the same document, you will encounter version conflicts. Use retry_on_conflict in the action metadata to let Elasticsearch handle the retry internally:

{ "update" : { "_index" : "logs", "_id" : "1", "retry_on_conflict": 3 } }
{ "doc" : { "status": "processed" } }

Verification and Rollback

To verify the result of a bulk operation, check the document count for the index:

GET /logs/_count

Rollback/Cleanup: Since the Bulk API changes the state of the index, the only way to "rollback" is to delete the documents by ID or delete the index if it was a fresh load:

DELETE /logs

Post-Load Step: Always restore your refresh interval to return the index to a searchable state:

PUT /logs/_settings
{
  "index": {
    "refresh_interval": "1s"
  }
}