Deploying Waku v2 Nodes with Relay and Store for Reliable Pub/Sub

The Problem: Message Loss in Decentralized Pub/Sub

Standard gossip protocols are efficient for real-time delivery but fail when a subscriber is offline or a network partition occurs. In a decentralized application, relying solely on a Relay (the protocol for real-time message forwarding) means any node that isn't connected at the exact moment of broadcast misses the data forever.

To solve this, Waku v2 implements a Store protocol. By deploying a node that combines both Relay and Store, you create a system where messages are not only gossiped in real-time but also cached by a subset of nodes, allowing offline peers to "catch up" on missed messages upon reconnection.

Prerequisites

• Go Environment: Go 1.21+ installed on the host machine.
• Waku Binaries: The go-waku binary (v0.20.x series) installed and available in your system PATH.
• Network Access: Open ports for the Waku protocol (default UDP/TCP ports) to allow peer discovery and gossip.
• Disk Space: Sufficient storage for the Store protocol's persistent database, as message caching consumes disk over time.

Node Configuration and Deployment

To ensure reliability, you must initialize the node with both the relay and store protocols enabled. While configuration files are available, using command-line flags is the most direct way to verify a deployment.

Run the following command on your server as a user with permissions to create directories in the specified data path:

go-waku --relay --store --mailserver --store-dir /var/lib/waku/store

Flag Definitions:

• --relay: Enables the real-time gossip protocol for low-latency message propagation.
• --store: Enables the caching protocol, allowing the node to save and serve historical messages.
• --mailserver: Enables the mailserver protocol, which helps in managing message delivery for specific peers.
• --store-dir: Specifies the local path for the persistent database. Risk: If this directory is deleted, the node loses its historical cache and cannot replay missed messages to peers.

Implementing a Content Topic and Messaging

Waku uses content topics to route messages. A topic is essentially a string that acts as a filter. For a production app, use a structured naming convention (e.g., /appname/version/category/topic) to avoid collisions.

Example: Publishing a Test Message
Use the waku-cli tool to send a payload to a specific topic. Ensure your payload is serialized (JSON or Protobuf) before sending, as Waku treats the payload as an opaque byte array.

# Run on Node A
waku-cli publish --topic "/myapp/1/msg/proto" --payload "{\"text\": \"Hello Waku!\"}"

Example: Subscribing to the Topic
On a second node, subscribe to the same topic to verify real-time delivery.

# Run on Node B
waku-cli subscribe --topic "/myapp/1/msg/proto"

Verifying Node Health and Reliability

A node is only reliable if it is well-connected to the mesh. You can monitor the internal state via the metrics endpoint (typically /:port/metrics).

Metric	Target Value	Indication
wakulink_relay_peers	≥ 6 peers	Healthy gossip mesh connectivity.
store_size_bytes	Growing/Stable	Messages are being cached successfully.
mailserver_queue_length	Low	Messages are being delivered without bottlenecking.

Testing the Store Recovery:

1. Start a subscriber node and confirm it is receiving messages.
2. Disconnect the subscriber node from the network (e.g., stop the process or block the port).
3. Publish several messages to the topic via a separate node.
4. Restart the subscriber node.
5. Check if the messages published during the downtime are delivered via the Store protocol.

Operational Limitations and Risks

• Payload Privacy: Waku does not encrypt payloads by default. All data sent over a content topic is visible to any node that subscribes to that topic. You must implement application-level encryption (e.g., AES or ECIES) before publishing.
• Disk Exhaustion: The Store protocol can consume significant disk space under high throughput. You must implement a pruning policy or monitor /var/lib/waku/store to prevent the node from crashing due to a full disk.
• Version Mismatch: Mixing nodes from different release series (e.g., pre-v0.20 and v0.20+) may lead to handshake failures. Ensure all nodes in your cluster use the same version.

Rollback Procedure

If the node deployment causes system instability or disk pressure, stop the process and remove the store directory to reset the state:

# Stop the process (e.g., via systemd or kill) 
# Remove the persistent store to reclaim space
rm -rf /var/lib/waku/store