Using HiveMQ Harvester to Capture and Replay MQTT Traffic for Testing

If you need realistic MQTT traffic for load tests, client debugging, or reproducing a production incident, writing synthetic publishers by hand rarely matches the real thing. HiveMQ's Harvester feature addresses this by capturing messages flowing through a broker and letting you replay them later against a test environment. This guide explains what Harvester does, walks through a representative configuration, and covers the limits that matter before you enable it anywhere near a busy broker.

What Harvester actually does

Harvester is a HiveMQ broker extension feature that persists a copy of MQTT publish messages to disk as they pass through the cluster. The captured data can then be replayed — republished to topics on the same or a different broker — so you can simulate real workloads in a staging environment, validate client behavior against realistic payloads, or debug an issue using the exact messages that triggered it.

Two design points shape everything else about it:

• Capture starts when you enable it. Harvester cannot recover messages published before it was active. If you are investigating a past incident, it will not help retroactively.
• It is intended for non-production use. Persisting every message adds I/O and CPU overhead to the broker, and captured data grows linearly with message volume. Treat it as a testing and diagnostics tool, not an always-on audit log.

A worked configuration

Harvester is configured in the broker's config.xml ( historically hive.xml, depending on your HiveMQ version — check which file your installation uses). A representative configuration looks like this:

<?xml version="1.0"?>
<hivemq>
    ...
    <harvester>
        <enabled>true</enabled>
        <data-directory>/opt/hivemq/harvester-data</data-directory>
        <retention>
            <max-age>P7D</max-age>
        </retention>
    </harvester>
</hivemq>

Key points about this example:

• <enabled> — Harvester is off by default and must be explicitly switched on. This is deliberate: it prevents accidental overhead on brokers that were never meant to capture traffic.
• <data-directory> — where captured messages are written. The operating system user running the HiveMQ process needs read and write permission on this path. Put it on a disk with headroom; a busy topic at a few thousand messages per second fills space quickly.
• <retention> — automatic cleanup of captured data older than the given period (ISO-8601 duration, so P7D is seven days). Without a retention limit, disk consumption is unbounded.

Element names and available options vary between HiveMQ versions, so treat the snippet above as a structural example and confirm the exact schema against the documentation for your installed version before restarting a broker with it. A configuration change requires a broker restart or reload to take effect, so plan for that on any shared system.

Replaying captured messages

Replay republishes the captured messages to their original topics, preserving payload content and, where supported, the original QoS level. Typical uses:

• Staging simulation: capture an hour of production traffic, replay it against a staging broker to validate a new client version or backend consumer under realistic load.
• Incident reproduction: capture traffic during a fault window, then replay it in isolation to find which message or sequence triggers the problem.
• Delayed replay: where a delay option is available, you can slow replay down to observe behavior that is invisible at full speed.

One caution: replaying to the same broker that still has live subscribers will deliver those messages to real clients. Always replay into an isolated environment, or at minimum a dedicated topic namespace, unless you are certain no production consumers will receive duplicates.

Limits and common mistakes

Performance overhead

Every captured message is an extra disk write on the broker's data path. On a lightly loaded broker this is negligible; on a broker approaching its throughput ceiling it can be the difference between keeping up and falling behind. Before relying on captured data for a load test, measure the broker's throughput with Harvester enabled versus disabled at a small scale so you know the overhead you are introducing.

Disk growth

Disk usage is roughly message rate × average payload size × retention period. For example, 1,000 messages per second at 1 KB each is about 86 GB per day. Set a retention period you can actually afford, and monitor the data directory during capture sessions rather than assuming cleanup will keep pace.

Expecting historical recovery

A recurring mistake is enabling Harvester after an incident and expecting to inspect the messages that caused it. Capture is forward-looking only. If post-incident forensics matter to you, Harvester needs to be running before the incident — which conflicts with the advice to keep it off production. The practical compromise is enabling it selectively on a pre-production environment that mirrors production traffic.

Edition and licensing assumptions

Feature availability differs between HiveMQ editions and versions. Confirm that your license and version include Harvester before building a test workflow around it.

Verifying it works

After enabling Harvester and restarting the broker, verify with a controlled test rather than waiting for real traffic:

1. Publish a known message with a unique payload to a test topic using any MQTT client (for example mosquitto_pub -h localhost -t harvester/test -m "probe-001", run from any host that can reach the broker; no special permissions beyond normal publish rights).
2. Check the configured data directory for newly written capture files.
3. Subscribe a client to the replay target in your isolated environment, run a replay, and confirm the payload, topic, and QoS match the original.
4. Watch disk usage of the data directory over the retention window to confirm cleanup runs as configured.

If the capture directory stays empty after publishing, check broker logs for permission errors on the data directory first — that is the most common cause.

Bottom line

Harvester is a practical way to turn real MQTT traffic into a reusable test asset, but only if you respect its constraints: enable it deliberately, bound retention to what your disk can hold, replay only into isolated environments, and validate the exact configuration schema against your HiveMQ version's documentation. Used that way, it replaces hand-written traffic simulation with something far more faithful to reality.