Reducing Noise in Waku Relay v2 with Content‑Addressed Deduplication

Problem: duplicate messages waste bandwidth

In a decentralized pub/sub network like Waku, every node forwards each message it receives. When the same payload is published multiple times—whether by accident or due to retry logic—each copy consumes bandwidth, storage, and CPU on every relay and subscriber. Traditional Waku relays have no built‑in way to recognize that two messages carry identical content, so the network can become noisy, especially in high‑frequency use cases such as transaction broadcasting or sensor streams.

Thesis: content‑addressed deduplication cuts redundant traffic while preserving privacy

Waku Relay v2 introduces a lightweight deduplication mechanism that works on the message payload itself. Each message includes a SHA‑256 hash of its payload; relays keep a Bloom filter of recent hashes. If a newly arrived message’s hash is already present in the filter, the relay treats it as a duplicate and does not rebroadcast it. Because the hash is derived only from the payload, end‑to‑end encryption remains intact—relays never see the plaintext.

How to enable deduplication

The feature is opt‑in. Run the relay binary with the --enable-dedup flag (or set the equivalent field in the Go library). No other configuration changes are required for basic operation.

waku-relay \
  --listen-address /ip4/0.0.0.0/tcp/30300 \
  --enable-dedup

Required permissions: the user must be able to execute the waku-relay binary and bind to the chosen network port (typically no special privileges needed for ports >1024).

Expected check: after the relay starts, its log should contain a line similar to deduplication filter initialized with size X. If you do not see this line, the flag may have been missed or the binary version may predate the feature.

Worked example: publishing the same transaction twice

Assume a Go client using the waku-go library. The client publishes a 1 KB JSON transaction payload two times in quick succession.

package main

import (
    "context"
    "log"
    "github.com/waku-org/go-waku/waku"
)

func main() {
    ctx := context.Background()
    relayAddr := "/ip4/127.0.0.1/tcp/30300/p2p/" // replace with actual peer ID
    wakuNode, err := waku.New(ctx, waku.WithRelayAddress(relayAddr))
    if err != nil {
        log.Fatalf("failed to create waku node: %v", err)
    }
    defer wakuNode.Close()

    payload := []byte(`{"tx":"0xabc...","value":123}`) // ~1 KB
    msg := &waku.Message{Payload: payload}

    // First publish
    if _, err := wakuNode.Publish(ctx, msg); err != nil {
        log.Printf("first publish error: %v", err)
    }

    // Second publish of identical payload
    if _, err := wakuNode.Publish(ctx, msg); err != nil {
        log.Printf("second publish error: %v", err)
    }
}

Where to run: on a developer workstation with Go 1.22+ and the go-waku module installed. Required permissions: none beyond normal user rights.

Expected behavior: the relay accepts the first message and rebroadcasts it to subscribers. When the second message arrives, the relay’s Bloom filter already contains its SHA‑256 hash, so the relay logs a duplicate detection event (e.g., duplicate detected for hash ) and does not forward the message. Consequently, each subscriber receives only one copy, saving roughly 1 KB of bandwidth per node in the network.

Practical verification: after running the example, inspect the relay’s output with waku-relay --stats. You should see an increase in the “duplicate hit” counter and no increase in the “message forwarded” counter for the second publish.

Trade‑off: memory usage and false‑positive rate

The Bloom filter trades memory for speed. A larger filter reduces the chance of a false positive (where a legitimate, unique message is mistakenly treated as a duplicate) but consumes more RAM. The current release hard‑codes a filter size optimized for moderate traffic; if you expect thousands of messages per second per relay, you may need to adjust the size via the --dedup-bits flag (available in newer builds) or upgrade to a version with a larger default.

Risk: a false positive could cause a valid message to be silently dropped, which might be unacceptable for certain financial or safety‑critical applications. Monitoring the filter’s hit/miss ratio via the stats command helps you tune the size before deploying to production.

Actionable closing

To start benefiting from deduplication today:

1. Download the latest Waku Relay v2 binary (v0.26.0 or newer).
2. Launch the relay with --enable-dedup and, if needed, --dedup-bits=<desired‑bits>.
3. Run your publisher(s) as usual; observe the relay logs for duplicate detection events.
4. Periodically check waku-relay --stats to ensure the false‑positive rate stays within your tolerance.

By enabling this feature, you cut unnecessary network traffic without sacrificing the privacy guarantees that make Waku suitable for decentralized messaging.