Reliable Atom Feed Polling with RFC 4287 Ids and RFC 5005 Paging

Why the classic polling loop can miss or duplicate entries

Many developers still rely on a naïve GET /feed + maxUpdated strategy. That approach assumes that every entry has a monotonically increasing atom:updated timestamp and that the server returns entries in strict order. In practice, servers batch updates, use the same timestamp for multiple edits, or drop ordering guarantees. The result is duplicate processing or, worse, missed entries.

Atom’s built‑in identity guarantees

RFC 4287 defines atom:id as a globally unique identifier for each entry. Unlike atom:updated, it never changes. A client can therefore store a set of seen atom:id values and filter duplicates reliably, even if the server reorders entries.

RFC 5005 paging: the missing link for incremental polling

RFC 5005 introduces atom:link elements with rel="next", rel="first", rel="last", and rel="archive". These links let a client traverse a feed page by page, rather than downloading the entire feed on every poll. A typical server might expose 10 entries per page and use rel="next" to point to the next set.

Practical polling strategy

1. Persist a cursor: store the maximum atom:updated value seen and a set of atom:id values.
2. Start at the first page: issue GET https://example.com/feed and follow the rel="first" link if present.
3. Loop over pages:

   while (linkRelNext) {
     const resp = await fetch(linkRelNext);
     const xml  = await resp.text();
     const doc  = new DOMParser().parseFromString(xml, "application/xml");
     const entries = Array.from(doc.querySelectorAll("atom\:entry", {namespaceURI: "http://www.w3.org/2005/Atom"}));
   
     for (const entry of entries) {
       const id   = entry.querySelector("atom\:id").textContent;
       const upd  = entry.querySelector("atom\:updated").textContent;
       if (seenIds.has(id)) continue; // dedupe
       if (upd <= cursor) continue;   // older than last processed
       process(entry);                // your business logic
       seenIds.add(id);
       if (upd > cursor) cursor = upd;
     }
     // find the next link
     const nextLink = doc.querySelector('atom\:link[rel="next"]');
     linkRelNext = nextLink ? nextLink.getAttribute("href") : null;
   }
   // After loop, persist cursor and seenIds
   

4. Use conditional GETs where the server supports ETag or Last‑Modified to avoid downloading unchanged pages. A 304 response means no new entries on that page.

Trade‑offs and limitations

• Clock skew: atom:updated is not guaranteed to be strictly increasing across servers; a late update may appear older than the cursor, causing a missed entry.
• Missing paging support: Not all Atom providers implement RFC 5005. Always inspect a page for rel="next" before assuming pagination.
• Storage overhead: Keeping a set of seen atom:id values grows with feed size. Periodically prune old IDs based on a retention policy.
• Pull‑only model: Atom has no push mechanism. Clients must poll at a suitable interval, balancing freshness against load.

Actionable checklist for a robust Atom poller

• Validate that the feed root contains <atom:feed> and that each entry has atom:id and atom:updated.
• On first run, store the atom:id set and the maximum atom:updated from the current page.
• On subsequent polls, start at rel="first" or the last known rel="next" link.
• For each page, dedupe by atom:id and skip entries older than the stored cursor.
• Persist the updated cursor and prune the ID set after a configurable retention window.
• Log any entries that fail to parse or lack required elements for manual inspection.

By combining atom:id deduplication with RFC 5005 paging, you can build a feed consumer that never reprocesses the same entry and never misses newer content, even on servers that batch or reorder updates.