Integrating Twitter API v2 Filtered Stream for Real‑Time Tweet Capture

Desired outcome

Set up a persistent connection to Twitter API v2’s filtered stream endpoint so that your application receives, in real time, only the tweets that match user‑defined rules.

Prerequisites

• A Twitter developer project with Elevated access (required for the filtered stream endpoint).
• API Key and API Secret Key from the project’s “Keys and tokens” tab.
• Ability to make HTTPS requests from your runtime (e.g., a Linux shell, Python script, or Node.js service).
• Network access to api.twitter.com on port 443 (no blocking proxies that strip headers).

Procedure

1. Obtain an OAuth 2.0 Bearer Token (app‑only flow)

Run the following command, replacing placeholders with your actual credentials. Execute it in a terminal where you have curl installed.

curl -X POST "https://api.twitter.com/oauth2/token" \
     -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" \
     -u "YOUR_API_KEY:YOUR_API_SECRET_KEY" \
     -d "grant_type=client_credentials"

Expected check: The response should be JSON with a field access_token. Copy that value; it is your Bearer Token. Treat it as a secret—do not log it or commit it to source control.

2. Define or update stream rules

Each rule consists of a value (the query string) and a tag (an identifier you choose). Use the Bearer Token from step 1 in the Authorization header.

curl -X POST "https://api.twitter.com/2/tweets/search/stream/rules" \
     -H "Authorization: Bearer YOUR_BEARER_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"add":[{"value":"#OpenAI lang:en","tag":"openai_test"}]}'

Expected check: HTTP 200 (or 201) with a JSON body containing the rule IDs under data. If you receive 429, you have exceeded the rule‑creation rate limit; wait and retry.

3. Open the filtered stream

Start a GET request that keeps the connection open. Include optional tweet.fields and user.fields to request extra metadata.

curl -X GET "https://api.twitter.com/2/tweets/search/stream?tweet.fields=created_at,author_id&user.fields=username" \
     -H "Authorization: Bearer YOUR_BEARER_TOKEN" \
     -N

The -N flag tells curl to output data as it arrives without buffering. Expected check: The first line of output should be a JSON object representing a tweet that matches your rule (e.g., contains "#OpenAI"). Subsequent lines arrive as new matching tweets are posted.

4. Process incoming lines

In your application code, read each line, parse it as JSON, and extract the fields you need. A minimal Python example:

import json, requests, os

BEARER = os.getenv('TWITTER_BEARER')
url = "https://api.twitter.com/2/tweets/search/stream?tweet.fields=created_at,author_id&user.fields=username"
headers = {"Authorization": f"Bearer {BEARER}"}

response = requests.get(url, headers=headers, stream=True)
response.raise_for_status()

for line in response.iter_lines():
    if line:
        try:
            tweet = json.loads(line)
            # Example extraction
            tweet_id = tweet.get('data', {}).get('id')
            text = tweet.get('data', {}).get('text')
            author_id = tweet.get('data', {}).get('author_id')
            print(f"{tweet_id}: {text} (by {author_id})")
        except json.JSONDecodeError:
            # Keep-alive line; ignore
            continue

Replace the print with your downstream logic (e.g., writing to a queue, storing in a database).

5. Error handling and reconnection

The filtered stream may close due to network issues, rate limits (HTTP 429), or Twitter‑initiated disconnects. Implement exponential backoff:

• On any non‑2xx response or socket error, close the current connection.
• Wait base * 2^attempt seconds (e.g., start at 1 s, then 2 s, 4 s, …) up to a maximum (e.g., 60 s).
• Reset the attempt counter after a successful connection.

Log the HTTP status code and any error message from Twitter’s payload to aid debugging.

Expected checks

• Bearer Token request returns access_token.
• Rule creation returns HTTP 200/201 with rule IDs.
• Stream connection yields JSON lines where data.text matches your rule.
• On simulated disconnection (e.g., kill the curl process), the client reconnects after the backoff interval.

Recovery options / Rollback

Adding rules modifies the stream’s filter set. To revert to a clean state, delete all rules:

# First, retrieve existing rule IDs
curl -X GET "https://api.twitter.com/2/tweets/search/stream/rules" \
     -H "Authorization: Bearer YOUR_BEARER_TOKEN"

# Assuming the response includes ids in data[].id, construct a DELETE payload
curl -X POST "https://api.twitter.com/2/tweets/search/stream/rules" \
     -H "Authorization: Bearer YOUR_BEARER_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"delete":{"ids":["ID1","ID2","ID3"]}}'

Expected check: HTTP 200 with a summary of deleted rules. After deletion, the stream will return no matches unless you add new rules.

Limitations and practical verification

• The filtered stream is subject to a rule‑count limit (≈25 active rules for Elevated access). Exceeding this limit returns HTTP 429 on rule‑add requests.
• Large or complex queries (many OR clauses) can increase latency; keep each rule’s value under 512 characters.
• To verify that backoff works, temporarily throttle your client by sending a rapid series of GET requests without waiting; you should observe HTTP 429 responses and see your client pause before retrying.
• Never expose the Bearer Token in logs, error messages, or client‑side bundles. Use environment variables or secret‑management services.

---

Following these steps will give you a reliable, real‑time pipeline for capturing tweets that match your criteria, with built‑in handling for authentication, rule management, stream consumption, and recovery from interruptions.