Architecture note for Twitter API v2 filtered stream

Problem and takeaway

You need a low‑latency, high‑throughput way to receive only the tweets that match your criteria from Twitter’s API v2. The takeaway is that a persistent HTTP/2 connection with chunked transfer, authenticated by a Bearer token and driven by a JSON rule payload, satisfies those needs while keeping trust boundaries clear.

Requirements

Low‑latency delivery of matching tweets, high throughput for many concurrent clients, reliable rule evaluation, and client‑driven reconnection with exponential backoff.

Minimal suitable design

A persistent HTTP/2 connection using chunked transfer encoding. The client opens the connection, sends a Bearer token in the Authorization header and a JSON payload describing filter rules. The server streams matching tweet objects as JSON lines, interleaved with occasional heartbeat messages to keep the connection alive.

Trust and data boundaries

Authentication occurs at the TLS layer; the server validates the Bearer token and treats all incoming rule updates as untrusted input, sanitizing them before they reach the tweet‑matching engine. Tweet payloads are considered trusted internal data but are still JSON‑parsed safely on the client side.

Operational checks

Clients should verify an HTTP 200 OK response, handle HTTP 429 (rate limit) and other 4xx/5xx errors, process heartbeat messages to detect silent disconnects, and implement jittered exponential backoff when reconnecting. Server‑side metrics track connection count, rule‑evaluation latency, and dropped events due to overload.

Failure modes

Network partitions cause connection drops; rule validation errors return HTTP 422 with details; excessive rule load can trigger HTTP 429; server‑side overload may close connections with a 503; malformed tweet JSON (unlikely) would cause client‑side parsing errors and should be logged and ignored.

Design‑change triggers

Moving to a gRPC‑based push service would replace the HTTP/2 chunked stream; adopting exactly‑once delivery would require durable client‑side checkpointing and server‑side acknowledgment; regulatory data‑residency rules could necessitate regional endpoints, altering the trust boundary and connection routing.

Example usage

Run the following command in a terminal where curl is available. Replace <YOUR_BEARER_TOKEN> with a token that has the tweet.read and offline.access scopes.

curl -N -H "Authorization: Bearer " -H "Content-Type: application/json" https://api.twitter.com/2/tweets/search/stream -d '{"add":[{"value":"twitter lang:en"}]}'

Expected checks: the response begins with HTTP/2 200, transfer‑encoding chunked, and delivers lines like {"data":{"id":"...","text":"..."}} separated by newlines, with occasional "" heartbeat lines. If the rule JSON is malformed, the server returns HTTP 422 with a descriptive error body, confirming validation at the trust boundary. If the network is disconnected, the client will stop receiving heartbeats and should initiate a jittered exponential backoff reconnect.

Limitations and practical verification

The filtered stream does not guarantee exactly‑once delivery; duplicate or missed tweets can occur during reconnections. Heartbeat messages are not a strict liveness guarantee—a missing heartbeat does not always mean failure, and a heartbeat can arrive while tweet delivery is stalled. To verify operation, watch the stream for matching tweet JSON lines and confirm that heartbeat lines appear roughly every 15‑30 seconds (the exact interval is undocumented but observable).