Automated Confluence Page Archiving to S3: Architecture, Checks, and Failure Handling

Problem Statement

Enterprise Confluence spaces accumulate pages that, after a configurable period, no longer require live editing but must remain accessible for compliance and audit purposes. The goal is to export those pages as PDFs to an external S3 bucket, flag them as archived inside Confluence, and keep all operations traceable. The solution must respect Confluence’s REST API limits, run on a schedule, and be resilient to transient failures.

Requirements

• Export pages older than a configurable threshold (e.g., 90 days).
• Stream the PDF export directly to an S3 bucket without storing it on the job host.
• Mark each processed page with a custom property (e.g., archived:true) to avoid duplicate work.
• Respect Confluence API rate limits and error codes.
• Maintain audit trails: Confluence audit logs, AWS CloudTrail, and internal metrics.
• Operate with minimal infrastructure: a scheduled function or lightweight service.
• Securely store API credentials and S3 access keys.
• Provide visibility into successes, failures, and retry counts.

Minimal Architecture

Confluence  ──┬── REST API (GET content, POST property, GET export/pdf) 
             │
            Scheduler (AWS Lambda or Spring Boot) ── streams PDF → S3
             │
            S3 Bucket (server‑side encryption, TLS in transit)

The core components are:

• Scheduler: Triggers nightly, queries pages by lastModified, and processes each page sequentially.
• Exporter: Calls /rest/api/content/{id}/export/pdf, streams the response body directly to S3 via the AWS SDK.
• Property Setter: After a successful upload, updates the page with a custom property to mark it archived.
• Monitoring: Exposes Prometheus metrics (e.g., export_success_total, export_failure_total) and sends alerts on persistent failures.

Trust & Data Boundaries

Component	Credential Scope	Data Flow
Confluence API Token	Read/write scope on target spaces only	Used exclusively by the scheduler to query and update pages
AWS IAM Role (Lambda)	S3 PutObject, CloudWatch Logs, CloudTrail	Uploads PDFs, writes logs, and records CloudTrail events
S3 Bucket Policy	Allow PutObject from the IAM role, deny public access	Stores PDFs, encrypted at rest (SSE‑S3 or SSE‑KMS)

Operational Checks

1. Rate‑Limit Monitoring: Track 429 responses; if exceeded, pause for retry-after seconds. Use exponential back‑off for subsequent attempts.
2. Retry Logic: Retry up to N=5 times per page. After exhausting retries, send the page ID to an SQS dead‑letter queue for manual investigation.
3. Metrics & Alerts: Expose Prometheus metrics; alert when export_failure_total exceeds threshold=3 in a 24‑hour window.
4. Audit Verification: Verify that each page has the archived:true property via a subsequent audit query.
5. Timeout Configuration: Set HTTP client timeout to 120s for the export call; fail fast if the response does not stream within this window.

Failure Modes & Mitigations

Failure	Impact	Mitigation
Network timeout to Confluence	Job stalls, page not archived	Retry with back‑off; after 5 attempts, move to DLQ
API quota exhaustion	Subsequent calls return 429	Pause for retry-after; throttle request rate to R=50 per minute
S3 upload error	PDF lost, page unarchived	Log CloudTrail; retry; if persistent, alert Ops
API token revoked	No further processing	Fail fast, generate alert, pause scheduler until token restored
Property update fails	Duplicate processing risk	Retry property update; if fails after retries, flag page for manual review

Design Change Triggers

• Confluence upgrades to a version that removes or changes the /export/pdf endpoint.
• Regulatory change requiring on‑prem archival storage instead of cloud.
• Page volume grows beyond the capacity of a single scheduled run (e.g., >10,000 pages per night). In that case, shift to a distributed worker model using ECS/Fargate or a Kinesis‑driven Lambda fleet.
• Audit policy demands stronger encryption or key‑management; update S3 SSE‑KMS configuration accordingly.

Example Implementation – Spring Boot + AWS SDK

Below is a concise snippet showing the core flow. Replace placeholders with actual values.

public void archivePage(String pageId, String spaceKey) throws IOException {
    // 1. Export PDF as a stream
    HttpHeaders headers = new HttpHeaders();
    headers.setBearerAuth(confluenceApiToken);
    headers.setAccept(Collections.singletonList(MediaType.APPLICATION_PDF));

    ResponseEntity response = restTemplate.exchange(
        String.format("https://%s.atlassian.net/wiki/rest/api/content/%s/export/pdf", confluenceHost, pageId),
        HttpMethod.GET,
        new HttpEntity<>(headers),
        Resource.class);

    InputStream pdfStream = response.getBody().getInputStream();

    // 2. Upload to S3
    String s3Key = String.format("archives/%s/%s.pdf", spaceKey, pageId);
    PutObjectRequest putReq = PutObjectRequest.builder()
            .bucket(s3BucketName)
            .key(s3Key)
            .contentType("application/pdf")
            .build();
    s3Client.putObject(putReq, RequestBody.fromInputStream(pdfStream, response.getHeaders().getContentLength()));

    // 3. Mark page as archived
    Map property = new HashMap<>();
    property.put("archived", true);
    HttpEntity<Map<String, Object>> propEntity = new HttpEntity<>(property, headers);
    restTemplate.postForEntity(
        String.format("https://%s.atlassian.net/wiki/rest/api/content/%s/property/archived", confluenceHost, pageId),
        propEntity,
        Void.class);
}

Key points:

• Use RestTemplate with a ClientHttpRequestFactory configured for a 120‑second timeout.
• Stream the PDF directly to S3 to avoid local disk I/O.
• Wrap the entire flow in a retry loop with exponential back‑off and a maximum of five attempts.
• After a successful upload, the property update is idempotent; if the property already exists, Confluence returns a 200 OK.

Verification Checklist

• Test Run: Execute the job against a test space; confirm PDFs appear in S3 and the archived property is set.
• Audit Log: Search Confluence audit stream for PROPERTY_CREATED events for the page.
• CloudTrail: Verify PutObject events for the bucket within the same timeframe.
• Metrics: Ensure export_success_total increments and no 429 spikes.
• Failure Simulation: Temporarily revoke the API token; observe retry behavior and that the job logs a failure after the final attempt.

Limitations & Caveats

• The solution assumes the Confluence instance exposes the /export/pdf endpoint; newer releases may require additional headers or a different URL pattern.
• Network latency must remain below the configured timeout; otherwise, the job will retry unnecessarily.
• Missing S3 permissions can cause silent failures if the SDK does not throw an exception; always enable logging for S3 actions.
• Audit logs are retained for 90 days by default; if your organization requires longer retention, adjust the Confluence audit log settings accordingly.
• For very large spaces, a single Lambda may hit concurrency limits; consider scaling out with multiple workers or ECS tasks.

Conclusion

By combining Confluence’s stable REST endpoints, a lightweight scheduler, and S3’s durable storage, you can automate compliance‑friendly page archiving with minimal infrastructure. The architecture is intentionally simple, but robust operational checks, clear trust boundaries, and defined failure modes ensure reliability. Monitor the system, validate against audit trails, and be ready to evolve the design when Confluence or regulatory requirements change.