Ingesting Real-Time Data with the Twitter API v2 Filtered Stream

The Challenge of Real-Time Ingestion

Building a real-time monitor for specific keywords or hashtags often leads to the mistake of polling a search endpoint repeatedly. This approach is inefficient and risks rapid rate-limiting. The Twitter API v2 Filtered Stream solves this by pushing matching tweets to your application via a persistent HTTP connection, ensuring you receive data the moment it is published.

Prerequisites

• A Twitter Developer account with access to the v2 API.
• A Bearer Token generated via the OAuth 2.0 Client Credentials flow. This token provides application-level access without requiring a specific user's authorization.
• A client capable of handling HTTP/1.1 chunked transfer encoding (most modern HTTP libraries support this).

Step 1: Defining Stream Rules

The filtered stream does not take query parameters in the connection request. Instead, you must define "rules" on the server side. These rules act as filters; only tweets matching these criteria will be pushed to your stream.

To add a rule, send a POST request to /2/tweets/search/stream/rules. Each rule must include a value (the query) and a tag (a unique identifier for your internal tracking).

# Example Request (using curl)
# Run this from your local terminal or server
curl -X POST "https://api.twitter.com/2/tweets/search/stream/rules" \
     -H "Authorization: Bearer <YOUR_BEARER_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"add": [{"value": "#engineering OR "system design", "tag": "tech_monitor"}]}'

Risk: If you attempt to add a rule that already exists or exceeds the maximum rule limit, the API will return an error. Always perform a GET request to the rules endpoint to audit current filters before adding new ones.

Step 2: Establishing the Connection

Once rules are set, you connect to the stream endpoint: GET /2/tweets/search/stream. Unlike a standard API call, this request does not close immediately. The server keeps the connection open and streams JSON objects as they arrive.

Implementation Requirements:

• Permissions: The Bearer Token must have read permissions.
• Connection Type: This is a standard HTTP/1.1 connection using chunked responses, not a WebSocket.
• Data Handling: You must process the incoming JSON stream asynchronously. If your application blocks the main thread while processing a tweet, the TCP buffer will fill up, and Twitter will terminate the connection.

Step 3: Managing Connection Failures

Streaming connections are fragile and will inevitably drop due to network instability or server-side resets. The most critical failure code to handle is HTTP 420 (Enhance Your Calm).

Status Code	Meaning	Required Action
429	Too Many Requests	Stop all connection attempts immediately. Wait for the rate limit window to reset.
420	Client Lag/Overload	Implement an exponential back-off. Ensure your data processing is offloaded to a queue (e.g., Redis or RabbitMQ).
5xx	Server Error	Retry connection after a short, randomized delay (jitter).

Example Diagnostic Decision

If your stream disconnects every 10 minutes without a specific error code, check your processing latency. If the time to process a single tweet exceeds the arrival rate of the tweets, the server will drop the connection to prevent buffer overflow. The solution is to move the tweet data into a buffer/queue and acknowledge the stream immediately, processing the data in a separate worker process.

Verification and Cleanup

To verify the stream is functioning, monitor your logs for the first 10 incoming JSON objects. Each object should contain the tweet text and the matching_rules field, which allows you to map the tweet back to the specific tag you defined in Step 1.

Rollback: Removing Rules

To stop the stream from filtering specific terms and clear your rule quota, send a DELETE request to the rules endpoint using the IDs returned during the GET or POST operations:

# Run this to remove specific rules
curl -X DELETE "https://api.twitter.com/2/tweets/search/stream/rules" \
     -H "Authorization: Bearer <YOUR_BEARER_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"delete": ["rule_id_12345"]}'

Limitations

• Concurrent Connections: A single developer account is limited to 100 concurrent stream connections.
• Rule Rotation: Rule IDs may change if rules are deleted and recreated; do not hardcode IDs in your long-term storage.
• Latency: While "real-time," there is a slight propagation delay between a tweet being posted and it appearing in the filtered stream.