Proxmox Ceph OSD Down or Out: A Diagnostic and Recovery Guide

You open the Proxmox web UI and see a Ceph OSD flagged as down or out, with placement groups (PGs) showing degraded or undersized. The useful takeaway: most OSD failures fall into three buckets — a crashed daemon, a failing disk, or a network partition — and you can distinguish them in a few commands before touching anything. Do not remove or reformat the OSD until you know which bucket you are in; an unnecessary removal triggers a full data rebalance and, done carelessly, can cost you data.

This guide assumes Proxmox VE 7.x/8.x with a hyperconverged Ceph cluster (Ceph Pacific or newer) and root or equivalent sudo access on the cluster nodes.

Recognizing the condition

Typical symptoms, visible in the Proxmox GUI under Node → Ceph → OSD or from any node's shell:

• OSD shows down (daemon not running) or out (cluster has stopped using it for data).
• ceph -s reports HEALTH_WARN or HEALTH_ERR with PGs in degraded, undersized, or incomplete states.
• VMs on Ceph-backed storage may still run — Ceph tolerates one OSD loss with the default size-3 replication — but redundancy is reduced.

Note the distinction: down means the process is unreachable; Ceph automatically marks an OSD out after a grace period (default 10 minutes, controlled by mon_osd_down_out_interval) and starts re-replicating its data elsewhere.

Cause and diagnostic quick reference

Symptom	Likely cause	Confirming evidence
OSD down, disk visible in lsblk, recent crash in log	OSD daemon crashed (bug, OOM, assertion failure)	Crash signature or assert in /var/log/ceph/ceph-osd.<id>.log
OSD down, disk missing from lsblk or throwing I/O errors	Disk hardware failure	dmesg I/O errors, SMART errors via smartctl -a /dev/sdX
Multiple OSDs on one node down simultaneously	Node or cluster-network partition	ping failures on the Ceph cluster network, heartbeat timeouts in logs
OSD flapping (down/up repeatedly)	Intermittent disk or network issue, or full disk	Repeated heartbeat failures; df -h on the OSD path near 95%+

Ordered checks

Run these from any Proxmox node in the cluster as root. Replace <id> with the failed OSD number (e.g., osd.3 → id 3).

1. Confirm cluster and OSD state.

   ceph -s
   ceph osd tree down

   ceph osd tree down lists exactly which OSDs are down and on which host. Note the host — you will run daemon-level commands there.
2. Check the daemon on the OSD's host.

   systemctl status ceph-osd@<id>

   If it is inactive (dead) or failed, the daemon stopped. If it is active but Ceph still reports the OSD down, suspect the cluster network instead.
3. Read the OSD log for a crash signature.

   tail -n 200 /var/log/ceph/ceph-osd.<id>.log

   Look for assert, segfault, heartbeat ... failed, or disk I/O errors. An assert or crash usually means a restart is worth trying; I/O errors point at the disk.
4. Check the physical disk. On the OSD's host:

   dmesg | grep -i -E 'error|fail' | tail -n 50
   smartctl -a /dev/sdX

   Replace /dev/sdX with the OSD's device (find it with ceph-volume lvm list or lsblk). Reallocated sectors or pending sectors mean the disk is dying — plan replacement, not repair.
5. Rule out the network. If several OSDs dropped together, verify the cluster network between nodes:

   ping <other-node-cluster-ip>

   Check that the cluster and public networks in /etc/pve/ceph.conf match your actual interfaces, and that no firewall change is blocking Ceph's ports (6800–7300 by default).

Fixes tied to findings

Daemon crashed, disk healthy

Restart the daemon on the OSD's host:

systemctl restart ceph-osd@<id>

Then mark the OSD back in if Ceph has already marked it out and it does not return on its own:

ceph osd in <id>

Watch ceph -s — PGs should begin recovering toward active+clean.

Disk failing

Do not restart into a dying disk. Replace the OSD properly so Ceph rebalances data first. From any node:

ceph osd out <id>

Wait until ceph -s shows all PGs active+clean (data has been re-replicated off the OSD). Only then, on the OSD's host:

systemctl stop ceph-osd@<id>
ceph osd purge <id> --yes-i-really-mean-it

Replace the physical disk, then recreate the OSD from the Proxmox GUI (Node → Ceph → OSD → Create) or with pveceph osd create /dev/sdX. Risk: running purge before rebalancing completes can permanently lose data if other replicas are also degraded. Verify active+clean first — this is the single most important check in this guide.

Network partition

Fix the underlying connectivity (cable, switch, MTU mismatch, firewall rule). Once the cluster network is stable, down OSDs typically rejoin automatically. If an OSD was marked out during the partition, run ceph osd in <id> after connectivity is confirmed.

Daemon crashes again after restart

A repeat crash with the same assert usually indicates corrupted OSD data or a version bug. Check for a pending Ceph package update (apt list --upgradable | grep ceph) and confirm all nodes run the same Ceph version (ceph versions). If versions match and the crash persists, treat the OSD as failed and follow the replacement path above.

Verification

1. ceph -s — all PGs return to active+clean and health returns to HEALTH_OK. Recovery may take minutes to hours depending on data volume.
2. ceph health detail — confirm no residual warnings such as OSD_DOWN or slow ops.
3. ceph osd stat — confirm the expected count of OSDs is up and in.
4. Monitor for at least 15 minutes: watch ceph osd stat or the GUI, watching for flapping that would indicate an unresolved hardware or network fault.

Escalation criteria

Stop and escalate (to your storage vendor, the Ceph community, or Proxmox support) when any of the following applies:

• PGs are stuck in incomplete or inactive — this means Ceph cannot find a complete copy of some data. Do not purge or restart anything further.
• More OSDs are down than your pool's replication can tolerate (for size 3 / min_size 2, losing two OSDs holding the same PG risks unavailability).
• Quorum is lost (ceph -s hangs or reports no quorum) — restarting monitors or OSDs without a plan can worsen this.
• Repeated crashes across multiple OSDs after an upgrade, suggesting a version regression.

The safe default throughout: observe first, rebalance before removing, and verify active+clean before any destructive step.