Twitter API v2 Filtered Stream: Rules, Connection Handling, and Production Gotchas

The Problem: Real-Time Tweet Filtering at Scale

You need to capture public tweets matching specific criteria — keywords, languages, media types, or conversation threads — as they happen. Polling the recent search endpoint wastes quota and adds latency. The filtered stream endpoint (GET /2/tweets/search/stream) pushes matching tweets over a persistent HTTP connection, but it requires a separate rule-management step and careful connection lifecycle handling.

Useful Takeaway

Create rules via POST /2/tweets/search/stream/rules, then open a single long-lived stream connection with the Bearer Token from an Elevated (or higher) access tier. Treat the connection as fragile: implement exponential backoff reconnection, deduplicate tweets by id, and monitor the ~20-second keep-alive newline to detect silent drops.

Worked Example: From Rule Creation to Live Stream

1. Create a Rule

Run this from a shell with curl and a valid Bearer Token (app-only OAuth 2.0). The token must belong to a project with Elevated or Academic access — Essential tier only permits the sampled stream.

TOKEN="YOUR_BEARER_TOKEN"
curl -X POST "https://api.x.com/2/tweets/search/stream/rules" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"add": [{"value": "lang:en (cat OR dog) -is:retweet", "tag": "pets-en"}]}'

Expected response includes a data array with the created rule's id and tag. Save the id — you need it to delete the rule later. A 400 error means syntax trouble; the response won't tell you which rule in a batch failed, so add rules one at a time during development.

2. Open the Stream

curl -N -H "Authorization: Bearer $TOKEN" \
  "https://api.x.com/2/tweets/search/stream?tweet.fields=created_at,public_metrics,lang&expansions=author_id,referenced_tweets.id"

The -N flag disables output buffering so you see each newline-delimited JSON object as it arrives. Each object contains a data field (the tweet) and a matching_rules array with the tag and id of the rule that matched. The includes object hydrates user and referenced tweet data requested via expansions.

3. Handle Reconnection with Backoff

Wrap the stream call in a loop that respects the keep-alive signal. Pseudocode:

backoff = 2  # seconds
while true:
    open stream connection
    for each line in response:
        if line is empty:  # keep-alive newline
            backoff = 2  # reset on healthy signal
            continue
        parse JSON, process tweet
    # connection dropped
    sleep(backoff)
    backoff = min(backoff * 2, 60)

Missing keep-alive for ~40 seconds indicates a silent disconnect — don't wait for TCP timeout.

Rule Syntax and Limits

• Length: 512 characters max per rule value.
• Operators: lang:, is:retweet, has:media, has:images, has:videos, entity:urls, context:domain, conversation_id:, from:, to:, @mention, place:, bounding_box: (requires place_id from Geo API).
• Removed in v2: near:, geocode: — use place-based operators instead.
• Tag: optional string (max 64 chars) returned in matching_rules to identify which rule matched.

Rate Limits and Connection Constraints

Operation	Limit	Window
Rule management (POST/GET/DELETE /rules)	50 requests	15 minutes
Stream connection	1 per app (Essential/Elevated) or per project (Academic)	Concurrent

Opening a second stream connection terminates the first. Design your client to hold exactly one connection per credential set.

Deleting Rules

There is no "delete all" endpoint. Retrieve existing rules, collect their IDs, then delete by ID list:

# 1. List rules
curl -H "Authorization: Bearer $TOKEN" \
  "https://api.x.com/2/tweets/search/stream/rules"

# 2. Delete by IDs (example IDs shown)
curl -X POST "https://api.x.com/2/tweets/search/stream/rules" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"delete": {"ids": ["1234567890123456789", "9876543210987654321"]}}'

Common Mistakes and How to Avoid Them

Batching Rules Without Validation

Sending multiple rules in one add array obscures which rule caused a 400. Add rules individually during development; batch only after each rule validates.

Ignoring Deduplication

The stream does not guarantee exactly-once delivery. After reconnect, you may receive tweets already processed. Deduplicate client-side using tweet.id (a 64-bit integer string).

Assuming Ordering

Tweets may arrive out of chronological order. If your pipeline requires ordering, buffer and sort by created_at.

Using Essential Tier for Filtered Stream

Essential access ($100/mo as of 2024) includes filtered stream but with lower monthly tweet caps. Verify current caps at developer.x.com/en/products/x-api before committing to production volume.

Missing Keep-Alive Handling

A silent disconnect (no TCP FIN, no data) leaves the client waiting indefinitely. Treat absence of the newline keep-alive for >40 seconds as a disconnect and trigger reconnection logic.

Verification Checklist Before Production

1. Create a test rule with a narrow query (e.g., from:your_test_account) and confirm tweets appear in the stream.
2. Simulate a network drop (kill the curl process) and verify your backoff loop reconnects and resumes receiving tweets.
3. Run the stream for 30 minutes and confirm keep-alive newlines arrive roughly every 20 seconds.
4. Check rule management rate-limit headers (x-rate-limit-remaining, x-rate-limit-reset) after a burst of rule changes.
5. Confirm your access tier's monthly tweet cap aligns with expected match volume.

Limitations to Plan For

• No historical replay — the stream only delivers tweets posted after the connection opens.
• No guaranteed delivery; network blips or backpressure can drop tweets.
• Rule changes take effect within seconds but are not instantaneous.
• Geo filtering requires a separate Geo API call to resolve place_id for place: or bounding_box: operators.