Optimizing Large File Uploads to IBM Cloud Object Storage with Multipart Uploads

When a single HTTP request just won’t cut it

In many production workloads you need to push files that are tens or even hundreds of gigabytes into IBM Cloud Object Storage (COS). A naïve PUT request can time‑out, consume a lot of memory, and can be aborted by transient network glitches. The result is a long, brittle upload that can cost you time and money.

IBM COS solves this with multipart upload. The idea is simple: split the file into independent parts, upload each part in parallel, and let the service stitch them together once all parts are finished.

Why multipart upload matters for engineers

• Speed: Parallel uploads reduce total transfer time, especially on high‑bandwidth links.
• Resilience: If a part fails, you only need to re‑upload that part instead of the whole file.
• Cost control: You can tune part size to balance request charges against memory usage.
• Scalability: The same code works for 10 MB files (no multipart) and 10 GB files.

Getting started with the Python SDK

First install the official SDK and create an IAM API key in the IBM Cloud console.

pip install ibm-cos-sdk

Now a minimal example that uploads a large file using multipart:

import os
from ibm_botocore.client import Config
from ibm_cos import CosClient

# Replace these placeholders
API_KEY = "YOUR_IAM_API_KEY"
COS_ENDPOINT = "https://s3.us.cloud-object-storage.appdomain.cloud"
BUCKET = "my-bucket"
FILE_PATH = "/path/to/large-file.bin"
OBJECT_NAME = os.path.basename(FILE_PATH)

# Create a client that talks to COS
client = CosClient(
    ibm_api_key_id=API_KEY,
    ibm_service_instance_id=BUCKET,
    ibm_cos_endpoint=COS_ENDPOINT,
    config=Config(signature_version='s3v4')
)

# Initiate a multipart upload – the SDK returns an upload_id
upload_id = client.create_multipart_upload(Bucket=BUCKET, Key=OBJECT_NAME)['UploadId']

# Split the file into 10 MB parts (default is 5 MB)
part_size = 10 * 1024 * 1024  # 10 MB
parts = []
with open(FILE_PATH, "rb") as f:
    part_number = 1
    while True:
        data = f.read(part_size)
        if not data:
            break
        response = client.upload_part(
            Bucket=BUCKET,
            Key=OBJECT_NAME,
            PartNumber=part_number,
            UploadId=upload_id,
            Body=data
        )
        parts.append({
            "PartNumber": part_number,
            "ETag": response['ETag']
        })
        part_number += 1

# Complete the multipart upload – this stitches all parts together
client.complete_multipart_upload(
    Bucket=BUCKET,
    Key=OBJECT_NAME,
    UploadId=upload_id,
    MultipartUpload={"Parts": parts}
)

print(f"Upload of {OBJECT_NAME} completed.")

Key points to check after running the script:

• Use client.list_objects(Bucket=BUCKET, Prefix=OBJECT_NAME) to confirm the object appears.
• Verify the size matches the original file: client.head_object(Bucket=BUCKET, Key=OBJECT_NAME)['ContentLength'].
• Attempt a GET request to ensure the object is readable.

Resuming a failed upload

If your client crashes after uploading some parts, the upload ID can be reused. Store the upload_id (e.g., in a database or a local file) before you start the upload. When you resume:

# List parts that have already been uploaded
existing_parts = client.list_parts(Bucket=BUCKET, Key=OBJECT_NAME, UploadId=upload_id)['Parts']
uploaded_part_numbers = {p['PartNumber'] for p in existing_parts}

# Re‑upload only the missing parts
with open(FILE_PATH, "rb") as f:
    part_number = 1
    while True:
        data = f.read(part_size)
        if not data:
            break
        if part_number in uploaded_part_numbers:
            part_number += 1
            continue  # Skip already uploaded part
        response = client.upload_part(
            Bucket=BUCKET,
            Key=OBJECT_NAME,
            PartNumber=part_number,
            UploadId=upload_id,
            Body=data
        )
        parts.append({
            "PartNumber": part_number,
            "ETag": response['ETag']
        })
        part_number += 1

After re‑uploading the missing parts, call complete_multipart_upload again. This approach avoids re‑transferring data you already sent.

Cost & performance trade‑offs

Part Size	Request Count (for 10 GB file)	Estimated Request Cost (US $)
5 MB (default)	2048	≈ 0.10
10 MB	1024	≈ 0.05
20 MB	512	≈ 0.025

Each part upload is a separate REST request, so larger part sizes reduce the number of requests and the associated cost. However, larger parts consume more memory on the client and may increase the time to recover from a failure (you’ll need to re‑upload a bigger chunk).

Eventual consistency and downstream processing

After complete_multipart_upload the new object is not immediately visible to all read operations. COS provides eventual consistency – a short delay (seconds to minutes) before the object can be retrieved by another request. If your pipeline requires the object to be immediately available, add a short polling loop or tag the object with a metadata key that downstream services watch.

What you must avoid

• Don’t attempt multipart upload for files smaller than 5 MB – the service will return InvalidPart errors.
• Never discard the upload_id if you want to resume; losing it means you must start over.
• Be cautious with extremely large part sizes (e.g., 100 MB) on low‑memory hosts; the SDK buffers the entire part before sending.

Actionable next steps

1. Run the provided example against a test bucket to confirm multipart works for your file size.
2. Instrument your upload code to log part numbers, ETags, and timestamps. This helps diagnose partial failures.
3. Set up Cloud Monitoring alerts for request count spikes on the bucket – a sudden increase can indicate a problematic upload.
4. Adjust part_size and max_concurrency (if using the SDK’s thread pool) to hit your performance sweet spot.
5. Implement a small persistence layer (e.g., a DynamoDB table) to store upload_id and part status for production use.

With these practices you can reliably move large files into IBM Cloud Object Storage, keep costs predictable, and recover gracefully from network hiccups.