Diagnosing SurrealDB Changefeed Subscription Failures

Recognizable condition

\n

A SurrealDB changefeed subscription appears to connect successfully (the client receives an acknowledgment) but no further data events are delivered, or the subscription terminates with an error.

\n\n

Cause and symptom table

\n\n\n\n\n\n\n\n\n\n\n\n

Cause	Typical symptom
Network interruption or firewall blocking WebSocket/HTTP used by changefeeds	Subscription connects but no data events are received after the initial acknowledgment.
Insufficient permissions (role lacks SELECT or LIVE QUERY privilege)	Subscription returns error ‘Permission denied’ or silently drops events.
Query syntax error in the LIVE SELECT statement	Server logs show ‘Invalid LIVE query’ and the subscription terminates.
Server version mismatch (client expects v1.x changefeed features, server is older)	Client receives ‘Unsupported operation’ or the connection is reset.
High server load causing changefeed buffer overflow	Initial events arrive, then lag increases and eventually stops; server logs show ‘Changefeed queue full’.

\n\n

Ordered diagnostic checks

\n

\n
1. Verify network connectivity to the changefeed endpoint\n
   \n
   • Where to run: any host that can reach the SurrealDB server (client machine or a diagnostic jump host).
   \n
   • Required permissions: none for the test; just ability to open a TCP socket.
   \n
   • Command (example): wscat -c ws://surreal-host:8000/rpc
   \n
   • Expected check: the WebSocket handshake succeeds and you can send a test frame.
   \n
   • Risk: none; if the connection fails, you have identified a network/firewall issue.
   \n
   \n
\n
2. Confirm the subscription role has required privileges\n
   \n
   • Where to run: SurrealDB SQL console (e.g., surreal sql) as a superuser or admin role.
   \n
   • Required permissions: ability to query the information_schema or run GRANT statements.
   \n
   • Command (example): SHOW ROLES FOR role_name; then check for SELECT and LIVE QUERY on the target table/namespace.
   \n
   • Expected check: output lists the needed privileges; if missing, proceed to fix.
   \n
   • Risk: granting privileges incorrectly can over‑expose data; follow least‑privilege principle.
   \n
   \n
\n
3. Validate the LIVE SELECT query syntax\n
   \n
   • Where to run: same SQL console as above, using the role that will subscribe.
   \n
   • Required permissions: SELECT on the table (already verified).
   \n
   • Command (example): LIVE SELECT * FROM table WHERE condition;
   \n
   • Expected check: server returns an initial snapshot and stays open; no ‘Invalid LIVE query’ in logs.
   \n
   • Risk: a malformed query will cause immediate termination; correct syntax resolves this.
   \n
   \n
\n
4. Check server version compatibility\n
   \n
   • Where to run: client side (to see client version) and server side (to see server version).
   \n
   • Required permissions: read access to version info (usually public).
   \n
   • Command (example): surreal version on client; SELECT version() on server.
   \n
   • Expected check: both report versions ≥ 1.0.0 if using changefeed features introduced in v1.x.
   \n
   • Risk: running an outdated server may lack features; upgrade is required.
   \n
   \n
\n
5. Monitor server load and changefeed queue status\n
   \n
   • Where to run: SurrealDB admin endpoint or logs.
   \n
   • Required permissions: ability to access /debug or view server logs.
   \n
   • Command (example): curl http://surreal-host:8000/debug/changefeed (if enabled) or grep logs for ‘Changefeed queue full’.
   \n
   • Expected check: queue length stays well below the configured limit; no recent overflow messages.
   \n
   • Risk: high load may require scaling or tuning; temporary back‑pressure is expected.
   \n
   \n
\n

\n\n

Fixes tied to findings

\n

\n
• Network/firewall: Ensure TCP port 8000 (default WebSocket) is open between client and server. Adjust security groups or iptables accordingly. After change, repeat the wscat test to confirm connectivity.
\n
• Insufficient permissions: Grant the minimal needed rights, e.g., GRANT SELECT, LIVE QUERY ON TABLE ns.table TO ROLE role_name;. Verify with SHOW GRANTS FOR ROLE role_name; then retry the subscription.
\n
• Query syntax error: Correct the LIVE SELECT statement. A valid example: LIVE SELECT * FROM ns.table WHERE field = $value;. Ensure any parameters are bound correctly. After fixing, the server should log ‘LIVE query started’ and return an initial snapshot.
\n
• Version mismatch: Upgrade the SurrealDB server to a release that includes live query support (≥ 1.0.0). If upgrading is not possible, use a client library version that matches the server’s capabilities.
\n
• Changefeed queue overflow: Reduce subscription load (e.g., add more specific WHERE clauses, limit number of concurrent changefeeds) or increase server resources (CPU, memory). If the server exposes a configuration for queue size, adjust it cautiously and monitor logs for overflow messages.
\n

\n\n

Escalation criteria

\n

If after completing all checks and applying the corresponding fix the subscription still fails to deliver events:

\n

\n
• Enable SurrealDB debug logging (set log.level = debug) and capture the full exchange during a subscription attempt.
\n
• Collect the client‑side trace (most drivers have a verbose mode) and server logs.
\n
• Contact SurrealDB support or open an issue on the official repository, providing:
\n
\n
• Server version and client library version.
\n
• Exact LIVE SELECT string used.
\n
• Network topology diagram (client ↔ server).
\n
• Relevant log snippets showing errors or ‘Changefeed queue full’ messages.
\n
\n

\n\n

Limitations and practical verification

\n

This guide covers the most common causes of changefeed subscription failures. It does not address:

\n

\n
• Bugs in the SurrealDB changefeed implementation that may require a patch.
\n
• Issues caused by client‑side application logic that incorrectly handles incoming messages.
\n
• Environment‑specific constraints such as proxy interference that alters WebSocket frames.
\n

\n

After any fix, a practical verification step is to open a test subscription with a known mutation:

\n

\n
1. In a separate terminal, insert a test record: CREATE ns.table SET test = true;
\n
2. Observe whether the test subscription receives the corresponding changefeed event within a few seconds.
\n
3. If the event appears, the subscription path is functioning; otherwise, repeat the diagnostic checks.
\n

\n