size vs. min_size: availability versus safety

Two different questions

size and min_size answer two separate questions about a replicated pool, and mixing them up is the single most common source of surprise once a Ceph cluster meets its first real failure. size is how many copies of every object Ceph tries to keep, spread across distinct failure domains. min_size is how many of those copies have to be reachable for Ceph to still accept writes at all. Data is never lost while at least one copy survives, but writes stop the moment fewer than min_size copies are reachable — that stoppage is a deliberate safety mechanism, not a bug: it exists specifically to avoid writing new data on top of a state Ceph can no longer be sure is consistent.

Proxmox’s own default is size 3 / min_size 2, and it is a good default: three copies survive one failed host with data still available, and even after that failure two copies still have to agree before Ceph accepts a write, so a second, simultaneous failure cannot silently corrupt data. Every other combination in this article trades away some part of that guarantee, usually for capacity.

The four common combinations

size / min_size combinations most often seen in the wild, and what a single host failure means for each.
size / min_sizecopies keptwrites stop whenwhat a single host failure does
3 / 23fewer than 2 copies reachablestill available (2 of 3 remain); no self-healing only if hosts = 3
3 / 13fewer than 1 copy reachablestill available, but a further failure can already accept writes on a single copy — the planner flags this danger
2 / 22fewer than 2 copies reachableIO on affected PGs blocks immediately until the host returns or recovery finishes
2 / 12fewer than 1 copy reachablestill available on the one surviving copy, but any further failure loses data outright

3 / 2 is the combination worth defaulting to unless there is a specific, understood reason to deviate. It tolerates one full host failure without interrupting writes, and it never accepts a write on fewer than two copies, so a second failure during recovery cannot corrupt already-written data — it can only pause IO on the affected placement groups until enough copies exist again.

Why min_size 1 is dangerous

The appeal of min_size 1 is that it keeps a cluster writable through almost any failure, and that is precisely the problem: it keeps accepting new writes onto a single, un-mirrored copy, so anything written during that window exists in exactly one place. A worn cable, a second disk failure, a node that does not come back — any one of those, arriving before the missing copies are rebuilt, permanently loses whatever was written in the meantime. Preferring a short, deliberate IO pause (what min_size 2 does) over a silent, permanent loss of new data is almost always the better trade for anything that holds real data.

size 2: no protection against a double failure

size 2 halves the raw capacity Ceph needs compared with size 3 for the same usable capacity, which is exactly why it is tempting on a small or budget-constrained cluster. The cost is that size 2 survives no double failure at all: with the recommended min_size 2, every single OSD failure already blocks IO on the placement groups it held, because only one copy remains and one is below the two required for writes. The planner marks size 2 as a warning for this reason, independently of whatever min_size is paired with it.

hosts = size: no self-healing

A replicated pool needs at least as many hosts as size under failure domain host, because CRUSH never places two copies of the same object on one host. Having exactly size hosts is enough to place the pool at all, but it is not enough for Ceph to heal itself: after one host fails, data stays available (the surviving copies still meet min_size), yet Ceph has nowhere left to recreate the missing copy — every remaining host already holds one. The cluster stays in a degraded state, with reduced redundancy, until the failed host is repaired or replaced, not until Ceph decides to act.

The nearfull, backfillfull and full ratios

Ceph watches raw OSD fill level against three fixed ratios, and none of them are configurable defaults the planner invented: mon_osd_nearfull_ratio at 0.85, mon_osd_backfillfull_ratio at 0.90, and mon_osd_full_ratio at 0.95 — see the Ceph configuration reference for the authoritative values.

ratiovalueCeph config keywhat happens
nearfull0.85mon_osd_nearfull_ratioCeph raises a health warning
backfillfull0.90mon_osd_backfillfull_ratioCeph refuses to backfill further data onto the OSD
full0.95mon_osd_full_ratioCeph blocks writes entirely

This is why the planner’s headline "planned" capacity figure is usable capacity multiplied by 0.85, not the raw usable figure itself: filling a cluster to 100 % of what it can technically hold means it was already past the point where Ceph starts warning, with no room left to even recover from a single failure without tripping backfillfull. Planning against the 85 % line, instead of the raw usable number, is what leaves headroom for exactly the kind of failure and recovery this article is about.

Worked examples

3 nodes, size 3 / min_size 2 — 3 nodes × 4 OSDs × 1.92 TB, size 3 / min_size 2: hosts = size, so a host failure leaves data available but Ceph cannot self-heal — plan for a fourth node.

3 nodes, size 2 / min_size 1 — 3 nodes × 1 OSD × 4 TB, size 2 / min_size 1: the planner raises danger for min_size 1 and warning for size 2 on the same pool at once.

Choosing a combination

  • Default to size 3 / min_size 2 unless there is a specific, understood reason not to.
  • Never run min_size 1 on data that matters — treat it as a temporary, supervised state at most.
  • If capacity pressure makes size 2 tempting, weigh it against the fact that every single OSD failure then blocks IO, not just a double failure.
  • Keep at least one more host than size requires so Ceph can self-heal without help.
size
Number of copies of every object a replicated pool keeps.
min_size
Minimum number of copies that must be reachable for Ceph to accept writes.

Open in the planner

Not affiliated with Proxmox Server Solutions GmbH, the Ceph Foundation or Red Hat/IBM. Proxmox and Ceph are trademarks of their respective owners.

Everything is calculated in your browser. Nothing is sent to a server.