Choosing the Right Offline Persistence for Waku Message Delivery

Problem and Decision

Waku nodes that operate with intermittent network connectivity must retain messages locally until the connection is re‑established. The chosen offline persistence mechanism determines whether messages survive a node restart, how quickly they are written, and what operational burden is introduced.

Constraints

• Durability across process restarts and power loss.
• Acceptable write latency under bursty traffic.
• Operational simplicity and maintenance overhead.
• Available resources such as disk space and CPU.
• Risk of data loss or corruption.

Supported Options

Option	Durability	Write Latency	Complexity	Operational Overhead
In‑memory queue	Low – lost on restart or power failure	Very low	Minimal	None
File‑based log (append‑only)	High – survives restarts, but can grow unbounded	Low‑moderate	Low	Requires log rotation policy
Embedded key‑value store (LevelDB/RocksDB)	High – transactional writes, crash‑safe	Low‑moderate	Medium‑high (column families, serialization)	Higher (library updates, backup)
External broker (Redis, RabbitMQ, etc.)	High – relies on broker availability	Moderate (network hop)	High (deployment, scaling)	Significant (monitoring, failover)

Trade‑offs

The in‑memory queue offers the fastest write and read paths, but any interruption eliminates stored messages. File‑based logs are simple to implement; they guarantee that messages are not lost when the process stops, yet each write incurs a disk sync and the log can become large without rotation, increasing wear and risk of uncontrolled growth.

Embedded key‑value stores such as LevelDB or RocksDB provide a middle ground: they store data on disk with crash‑consistent transactions, delivering low latency comparable to files while avoiding unbounded growth. The trade‑off is added complexity—developers must manage column families, handle serialization of Waku messages, and ensure proper transaction boundaries to prevent corruption.

Relying on an external broker introduces network latency and a single point of failure if the broker becomes unreachable. It also adds operational overhead for provisioning, monitoring, and scaling the broker infrastructure, which may be unnecessary for a single‑node or small‑scale deployment.

Concrete Implementation Example

Below is a practical example that configures Waku to use LevelDB as the offline persistence layer. The steps assume you have a Waku node binary compiled with LevelDB support.

1. Create a dedicated data directory with write permissions for the Waku process.

   mkdir -p /var/lib/waku/leveldb
   chown -R waku_user:waku_user /var/lib/waku/leveldb
   

   Run the command with sudo if the user running Waku lacks write access to the directory.

2. Start the Waku node with the LevelDB persistence flag.

   waku-node --persistence leveldb --leveldb-path /var/lib/waku/leveldb
   

   Replace waku-node with the actual binary name and adjust the path placeholder /var/lib/waku/leveldb to match your environment.

3. Validate durability by stopping the node mid‑session and restarting it.
   1. Publish a test message while the node is running.
   2. Stop the Waku process (e.g., kill -SIGTERM <pid>).
   3. Start the node again using the same command.
   4. Check that the test message appears in the output or is delivered to the intended subscriber.

   If the message is re‑delivered, the LevelDB store is functioning correctly.

4. Monitor write latency and disk usage.

   # Example: measure average write latency over 1000 messages
   waku-bench --type write --count 1000 --persistence leveldb
   

   Compare the measured latency with the in‑memory baseline to confirm that LevelDB does not introduce unacceptable delays.

Verification Checklist

• Run a restart test as described; confirm message re‑delivery.
• Execute a synthetic traffic benchmark for each storage option and record write latency, message loss rate, and ordering consistency.
• Inspect the LevelDB directory for compacted files to ensure no corruption after repeated stops.
• If using a file‑based log, implement a rotation script (e.g., using logrotate) to prevent unbounded growth.

Choosing the right offline persistence mechanism for Waku hinges on balancing durability, performance, and operational cost. LevelDB offers a robust compromise for most deployments, while simpler file logs suit environments where disk wear is acceptable and complexity must be minimized. External brokers excel when scaling across multiple nodes but introduce network dependency. By following the concrete configuration and validation steps above, you can verify that your selected mechanism meets the intermittent‑connectivity requirements of your Waku deployment.