Waku Store Architecture: Designing for Peer-to-Peer Offline Retrieval

The Problem: Message Loss During Disconnection

In a pure peer-to-peer (P2P) gossip network, messages are transient. If a node is offline when a message is broadcast, that data is lost to them unless a mechanism exists to persist and retrieve historical messages without relying on a centralized server.

Requirements

• Offline Retrieval: Nodes must be able to query historical messages after a period of disconnection.
• Low-Latency Delivery: Real-time messaging must remain fast, with storage acting as a secondary persistence layer.
• Decentralized Integrity: No single trusted authority should be required to verify that a retrieved message is authentic.
• Partition Tolerance: Local storage must remain functional and queryable even when the node is isolated from the network.

Minimal Design

The smallest suitable implementation consists of a waku-store plugin integrated into the libp2p stack. This plugin intercepts incoming messages from the Waku Relay layer and persists them to a local key-value (KV) store, such as LevelDB.

Messages are indexed by two primary keys: the contentTopic (the specific channel or subject) and the timestamp. To retrieve data, the node implements a request-response protocol using StoreRequest and StoreResponse RPCs, allowing peers to ask for messages within a specific time range for a given topic.

Trust and Data Boundaries

To maintain security in an untrusted P2P environment, the architecture defines strict boundaries:

• Local Trust: The node trusts only its own local disk and the cryptographic signatures attached to messages.
• Peer Distrust: Data received from peers is treated as untrusted. The store verifies the message signature before persisting it to disk to prevent the storage of forged data.
• Topic Filtering: To prevent storage exhaustion from irrelevant data, the node only persists messages for contentTopics it has explicitly subscribed to.

Operational Checks

Maintaining a healthy store requires monitoring the intersection of disk I/O and network gossip:

1. Database Integrity: Periodically run a read-only scan or use tools like ldb repair to detect bit rot or corruption in the KV store.
2. Query Latency: Monitor the 95th percentile of StoreRequest response times. Spikes often indicate disk contention or inefficient indexing.
3. Disk Capacity: Track the storage directory size. Alerts should trigger at 80% capacity to prevent abrupt crashes.
4. Subscription Health: Verify that the gossip subscriptions for active topics are maintained; if a subscription drops, the store stops receiving new data for that topic.

Failure Modes

Failure Scenario	System Behavior	Mitigation
Disk Full	Store returns ERR_STORE_OOM; inbound gossip is throttled.	Implement storage quotas and automated pruning of old messages.
DB Corruption	Store plugin fails to initialize or crashes during reads.	Plugin restart attempt; fallback to Relay-only mode (no history).
Network Partition	Remote queries fail, but local data remains accessible.	Automatic resynchronization via gossip once connectivity is restored.
Request Spam	High CPU/Disk load due to malicious StoreRequest floods.	Per-peer rate limiting and libp2p peer-scoring penalties.

Example Configuration and Query

For a Go-Waku (v0.20.x) deployment, the store is enabled via the configuration file. Ensure the process has write permissions for the specified path.

# Go-Waku store configuration
store.enabled=true
store.db-path=/var/lib/waku/store
store.max-size-mb=4096
listen-addresses=/ip4/0.0.0.0/tcp/30303

Once the node is running, you can query the store via the waku-rpc endpoint using curl. This example requests messages for a specific topic from the beginning of time (timestamp 0) to a high future value:

curl -X POST http://127.0.0.1:8545/waku/v1/store \n-H 'Content-Type: application/json' \n-d '{"contentTopic":"/myapp/1/msg","timestampFrom":0,"timestampTo":9999999999}'

The resulting JSON array contains the payload and signature. The client must verify the signature against the sender's public key to ensure the message was not tampered with during storage.

Verification and Practical Checks

To verify the store is functioning correctly, perform the following sequence:

1. Persistence Check: Publish a message to a subscribed topic. Restart the Waku node. Issue the curl query above; the message should still be present.
2. Signature Validation: Compare the returned signature from the store query with the signature of the original published message. They must be identical.
3. Resource Exhaustion Test: Simulate a full disk by creating a large dummy file in the store partition (e.g., dd if=/dev/zero of=/var/lib/waku/store/fill bs=1M count=4000). Attempt to publish a new message and check the logs for ERR_STORE_OOM or throttled traffic.

Conditions That Would Change the Design

• Strong Consistency: If the system requires linearizable reads across multiple nodes, the local KV store would need to be replaced by a distributed consensus log (e.g., Raft).
• Verifiable Inclusion: If clients need proof that a message exists in the store without downloading the whole set, the design would shift to a Merkle-tree index to provide inclusion proofs.
• High-Volume Archiving: If storage costs exceed local disk capacity, a tiered architecture would be required, moving older messages to an object store (like IPFS) while keeping pointers locally.

Limitations

The store relies on an honest majority for gossip propagation; if malicious peers eclipse a topic, messages may never reach the store. Additionally, persistence introduces a disk I/O bottleneck that does not exist in pure relay nodes, requiring careful capacity planning.