Does k3os reapply boot_cmd and run_cmd idempotently on every boot, or can retries duplicate their writes?

Short answer

Yes — k3os executes boot_cmd and run_cmd unconditionally on every boot and on every config reapply. There is no built-in deduplication, no marker file, and no "already ran" state for command fields. The declarative convergence you see with hostname, SSH keys, or write_files applies only to structured config fields; anything you put in a command field is just a shell snippet k3os runs, and its side effects are entirely your responsibility.

What k3os actually tracks

k3os's config machinery converges the system toward the declared state for fields it understands structurally. Reapplying the same SSH authorized keys doesn't duplicate them because k3os writes the target state, not an append log. Command fields don't get that treatment: they are opaque strings executed at the appropriate stage (boot_cmd early in boot, run_cmd after the system is up). k3os records which config version was applied, but that bookkeeping does not gate command execution — a reapply of the identical config will still re-run your commands.

So to your three questions directly:

• Every boot/apply, no dedup: correct. Retries after a failed upgrade or a manual reapply will execute the commands again.
• Self-idempotency is the intended pattern: yes. Treat these fields like cloud-init runcmd without the once-per-instance semantics — write them as if they will run an unbounded number of times.
• Marker/state file: none is documented for command steps. If you want run-once behavior, you have to implement the marker yourself.

Making your commands safe

For the side effects you describe, the standard guards are enough:

boot_cmd:
  - mkdir -p /var/lib/myapp/data
  - grep -q '^fs.inotify.max_user_watches' /etc/sysctl.conf || \
      echo 'fs.inotify.max_user_watches=524288' >> /etc/sysctl.conf
run_cmd:
  - '[ -f /var/lib/myapp/.provisioned ] || { /usr/local/bin/provision.sh && touch /var/lib/myapp/.provisioned; }'

The last pattern — a sentinel file you create on success — is the closest equivalent to a "marker file" you asked about, and it's the right tool for any step that genuinely can't be made idempotent (a one-time database seed, a certificate enrollment).

Verifying the behavior on your nodes

1. Add touch /var/lib/rancher/k3os/rerun_test_$(date +%s) to run_cmd.
2. Force a config reapply, then list /var/lib/rancher/k3os/rerun_test_*. Multiple timestamped files confirm unconditional re-execution.
3. Check journalctl -u k3os for repeated execution of your command entries across the reapply.

Remove the test command afterward.

One caveat

This reflects k3os's behavior as of its final releases; since the project is deprecated, there is no upstream change coming, but if you are running a fork or a pinned older build, the sentinel-file test above is the reliable way to confirm the semantics on your exact version. Either way, writing the commands idempotently is correct under both interpretations — it costs nothing if k3os ever did dedupe, and it's the only thing protecting you if it doesn't.