Automating Confluence at Scale: Practical Patterns with the Cloud REST API v2

The problem: bulk content changes without hitting limits

Teams that manage hundreds of Confluence pages — release notes, runbooks, architecture docs — quickly discover that the old REST v1 endpoint forces a request per page and offers no pagination metadata. When a CI/CD job tries to create or update 200 pages in a single run, the Cloud tenant returns 429 Too Many Requests after the first 100 calls, and the pipeline stalls. The v2 API (released 2023) solves the pagination and field‑selection gaps, but it introduces a strict per‑app rate limit (100 requests/minute) and a new OAuth 2.0 (3LO) token lifecycle that many scripts still ignore.

Thesis

Adopt the v2 /content endpoint with body-format=storage, batch requests using the expand parameter, and implement a token‑aware rate‑limit wrapper. This yields idempotent, observable bulk operations that work in both Cloud and Data Center (where the same endpoint exists but limits are configurable).

Key patterns

1. Authenticate once, refresh atomically

OAuth 2.0 (3LO) access tokens expire after one hour; refresh tokens rotate and invalidate the previous one. Store the token pair in a secret manager (e.g., Vault, GitHub Actions secrets) and update both fields in a single transaction.

# Example: token refresh in a Bash step (run on CI runner)
curl -X POST https://auth.atlassian.com/oauth/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=refresh_token" \
  -d "client_id=${CLIENT_ID}" \
  -d "client_secret=${CLIENT_SECRET}" \
  -d "refresh_token=${REFRESH_TOKEN}" | \
  jq -r '.access_token, .refresh_token' > /tmp/tokens.new
# Atomic replace
mv /tmp/tokens.new ${TOKEN_FILE}

Permissions: the OAuth client must have scopes read:confluence-content.all write:confluence-content.all. Risk: if the refresh call fails, the pipeline loses write ability until a manual re‑authorisation.

2. Respect the 100‑req/min ceiling with a retry‑after loop

Every authenticated response includes X-RateLimit-Limit, X-RateLimit-Remaining and, on 429, a Retry-After header (seconds). A simple wrapper reads the header and sleeps before the next call.

function v2_post() {
  local url=$1; shift
  local payload=$1
  while true; do
    resp=$(curl -s -w "\n%{http_code}" -X POST "${url}" \
      -H "Authorization: Bearer ${ACCESS_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "${payload}")
    http_code=$(tail -n1 <<<"${resp}")
    body=$(sed '$d' <<<"${resp}")
    if [[ ${http_code} -eq 429 ]]; then
      retry=$(curl -s -I -X POST "${url}" -H "Authorization: Bearer ${ACCESS_TOKEN}" | grep -i retry-after | awk '{print $2}' | tr -d '\r')
      sleep "${retry:-60}"
      continue
    fi
    echo "${body}"
    return
  done
}

Where to run: any CI/CD agent with network access to api.atlassian.com. Check: after each call, verify X-RateLimit-Remaining > 0; if it drops below 5, proactively sleep 30 s.

3. Reduce round‑trips with expand and body-format=storage

A single GET /wiki/api/v2/content?expand=body.storage,version,ancestors&body-format=storage&limit=50 returns storage‑format XHTML, the current version number, and the parent chain — enough to decide whether a page needs an update or a create. The storage format is the same XHTML used by the editor, so you can manipulate it with an XML parser (e.g., xmllint or Python lxml) before sending a PUT.

Worked example: bulk‑publish release notes from a markdown directory

Assume a repo folder release-notes/ with one .md file per version. The pipeline converts each file to storage‑format XHTML, then creates or updates the corresponding page under space DOCS.

1. Convert markdown → storage XHTML (run on the build agent):

   pandoc -f markdown -t html --wrap=none release-notes/1.2.3.md | \
     sed 's/
   /<\/ac:plain-text-body><\/ac:structured-macro>/g' > /tmp/1.2.3.storage

2. Check existing page (uses expand=version to get current version number):

   existing=$(v2_get "https://api.atlassian.com/ex/confluence/${CLOUD_ID}/wiki/api/v2/content?title=1.2.3&space-key=DOCS&expand=version&body-format=storage")
   page_id=$(echo "${existing}" | jq -r '.results[0].id // empty')
   version=$(echo "${existing}" | jq -r '.results[0].version.number // 0')

3. Create or update (idempotent PUT with If-Match header):

   payload=$(jq -n --arg body "$(cat /tmp/1.2.3.storage)" --argjson ver $((version+1)) \
     '{title:"1.2.3", type:"page", space:{key:"DOCS"}, body:{storage:{value:$body, representation:"storage"}}, version:{number:$ver}}')
   if [[ -n ${page_id} ]]; then
     etag=$(v2_head "https://api.atlassian.com/ex/confluence/${CLOUD_ID}/wiki/api/v2/content/${page_id}" | grep -i etag | awk '{print $2}' | tr -d '\r')
     curl -X PUT "https://api.atlassian.com/ex/confluence/${CLOUD_ID}/wiki/api/v2/content/${page_id}" \
       -H "Authorization: Bearer ${ACCESS_TOKEN}" \
       -H "Content-Type: application/json" \
       -H "If-Match: ${etag}" \
       -d "${payload}"
   else
     v2_post "https://api.atlassian.com/ex/confluence/${CLOUD_ID}/wiki/api/v2/content" "${payload}"
   fi

Verification: after the loop, query GET /wiki/api/v2/content?space-key=DOCS&expand=version and confirm each page’s version.number matches the expected increment. Limitation: pages larger than 64 KB omit the body in webhook payloads; if you rely on webhooks for downstream sync, fall back to a scheduled poll using the since parameter.

Trade‑offs and gotchas

• Rate‑limit sharing: Cloud sandbox tenants that reuse the same OAuth client ID share the 100 req/min quota with production. Register a distinct OAuth app per environment.
• Macro IDs: Storage XHTML contains ac:macro-id values that must be regenerated on Cloud import; hard‑coded IDs cause duplicate‑macro errors. Strip or regenerate them during the markdown‑to‑storage step.
• Data Center direct DB reads: Querying SPACEPERMISSIONS directly voids Atlassian support. Use the REST /space/{key}/permission endpoint for audit scripts.
• Webhook ordering: Cloud webhooks (content_created, content_updated) have no guaranteed order and retry with exponential backoff. Design consumers to be idempotent and to fetch the latest page state via the content endpoint.

Actionable next steps

1. Create a dedicated OAuth 2.0 (3LO) client for each environment and store client secret + refresh token in your secret manager.
2. Add a reusable rate‑limit wrapper (like the v2_post function) to your shared CI library.
3. Migrate existing bulk scripts to use /wiki/api/v2/content with expand=body.storage,version,ancestors and body-format=storage.
4. Implement a nightly verification job that lists all pages in target spaces and asserts version numbers match the source of truth.
5. Monitor the Atlassian Developer Changelog monthly for v2 deprecations (e.g., expand=metadata.labels removal) and adjust scripts accordingly.

By treating the v2 API as a first‑class contract — authenticating cleanly, paging deliberately, and respecting the hard rate limit — you turn Confluence from a manual wiki into a programmable knowledge base that keeps pace with your release cadence.