Solving Payload Bloat: Lessons in API Contract Design from LinkedIn's Scale

The Cost of the 'Everything' Payload

When building a professional network or a content feed, the instinct is often to create a comprehensive /profile or /user endpoint. This endpoint returns everything: name, headline, work history, education, skills, and recent activity. While convenient for a developer during a sprint, this approach creates a performance bottleneck as the product grows. Over-fetching—sending data the client doesn't actually need for a specific view—increases latency, drains mobile battery, and puts unnecessary pressure on the database.

The technical challenge is balancing a stable API contract with the need for flexible, lean data delivery. The goal is to move from fixed payloads to a client-driven selection model where the requester defines the shape of the response.

Moving Toward Explicit Field Selection

Instead of versioning endpoints (e.g., /v1/profile, /v2/profile) every time a new field is added, a more scalable engineering decision is to implement field selection. This allows the client to specify exactly which attributes it requires for the current UI state.

By treating the API as a set of contracts rather than fixed files, you achieve three things:

• Reduced Payload Size: A "mini-profile" for a comment section doesn't need the user's full employment history.
• Backward Compatibility: New fields can be added to the backend schema without breaking old clients, as those clients simply don't request the new fields.
• Optimized Backend Resolvers: The server can skip expensive database joins or third-party API calls if the requested fields don't require that data.

Managing Large Datasets with Cursor Pagination

For feeds and activity lists, traditional offset pagination (LIMIT 10 OFFSET 100) fails at scale. In a high-velocity environment, new items are inserted constantly. If a user is on page one and a new post is published, the first item of page two becomes a duplicate of the last item of page one.

The solution is cursor-based pagination. Instead of a page number, the server returns a next cursor—an opaque string representing the position of the last item retrieved. The client sends this cursor back to fetch the next batch. This ensures that the window of data remains stable regardless of new insertions.

Worked Example: Fixed vs. Selected Payloads

Consider a scenario where a client needs to display a user's basic info and their last few posts. Below is a comparison of how these requests are handled in a mock implementation of a field-selection API.

The Over-fetched Approach (Fixed)

# Request
GET /api/profiles/123

# Response (contains 50+ fields, most unused)
{
  "id": "123",
  "name": "Jane Doe",
  "headline": "Software Engineer",
  "bio": "Long biography...",
  "work_history": [...], 
  "education": [...],
  "skills": [...],
  "settings": {...},
  "activity": [...]
}

The Selected Approach (Contract-based)

In this model, the client uses a fields query parameter to specify the required data and a limit for the nested activity slice.

# Request
GET /api/profiles/123?fields=name,headline,activity(limit:3)

# Response (Lean and specific)
{
  "id": "123",
  "name": "Jane Doe",
  "headline": "Software Engineer",
  "activity": [
    {"id": "post_1", "text": "Hello world"},
    {"id": "post_2", "text": "Learning API design"},
    {"id": "post_3", "text": "Scale is hard"}
  ],
  "pagination": {
    "next_cursor": "dXNlcl8xMjNfcmVjb3JkXzQ1"
  }
}

The Trade-off: Complexity Shift

Moving to a flexible contract model isn't free. It shifts the complexity from the endpoint definition to the query validation layer. When clients can request arbitrary fields, the server must implement:

• Cost Analysis: Preventing "expensive" queries (e.g., requesting 10 nested levels of connections) that could crash the database.
• Field-Level Authorization: Ensuring a user can request their own email but not someone else's, even if both are part of the same Profile object.
• Caching Challenges: You can no longer cache a single /profile/123 response, as every client might request a different subset of fields. Caching must happen at the object or field level.

Verification and Implementation

To verify if your API is suffering from over-fetching, monitor the ratio of bytes sent to bytes utilized by the frontend. If your profile payload is 100KB but the UI only renders 2KB of text, you are a candidate for field selection.

When implementing this, start by introducing a fields parameter to a single high-traffic endpoint. Validate the requested fields against a whitelist of "safe" attributes before passing the request to the data layer to prevent unauthorized data exposure.