After a crushmap is changed (e.g. addition/removal of devices, modification of weights or tunables), objects may move from one device to another. The crush compare command can be used to show what would happen for a given rule and replication count. In the following example, two new OSDs are added to the crushmap, causing 22% of the objects to move from the existing OSDs to the new ones.
$ crush compare --rule firstn \
--replication-count 1 \
--origin before.json --destination after.json
There are 1000 objects.
Replacing the crushmap specified with --origin with the crushmap
specified with --destination will move 229 objects (22.9% of the total)
from one item to another.
The rows below show the number of objects moved from the given
item to each item named in the columns. The objects% at the
end of the rows shows the percentage of the total number
of objects that is moved away from this particular item. The
last row shows the percentage of the total number of objects
that is moved to the item named in the column.
osd.8 osd.9 objects%
osd.0 3 4 0.70%
osd.1 1 3 0.40%
osd.2 16 16 3.20%
osd.3 19 21 4.00%
osd.4 17 18 3.50%
osd.5 18 23 4.10%
osd.6 14 23 3.70%
osd.7 14 19 3.30%
objects% 10.20% 12.70% 22.90%
The crush compare command can also show the impact of a change in one or more “tunables”, such as setting chooseleaf_stable to 1.
$ diff -u original.json destination.json
--- original.json 2017-03-14 23:41:47.334740845 +0100
+++ destination.json 2017-03-04 18:36:00.817610217 +0100
@@ -608,7 +608,7 @@
"choose_local_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
- "chooseleaf_stable": 0,
+ "chooseleaf_stable": 1,
"chooseleaf_vary_r": 1,
"straw_calc_version": 1
}
In the following example some columns were removed for brevity and replaced with dots. It shows that 33% of the objects will move after chooseleaf_stable is changed from 0 to 1. Each device will receive and send more than 1% and less than 3% of these objects.
$ crush compare --origin original.json --destination destination.json \
--rule replicated_ruleset --replication-count 3
There are 300000 objects.
Replacing the crushmap specified with --origin with the crushmap
specified with --destination will move 99882 objects (33.294% of the total)
from one item to another.
The rows below show the number of objects moved from the given
item to each item named in the columns. The objects% at the
end of the rows shows the percentage of the total number
of objects that is moved away from this particular item. The
last row shows the percentage of the total number of objects
that is moved to the item named in the column.
osd.0 osd.1 osd.11 osd.13 osd.20 ... osd.8 osd.9 objects%
osd.0 0 116 180 0 3972 ... 138 211 1.89%
osd.1 121 0 129 64 116 ... 112 137 1.29%
osd.11 194 126 0 12 0 ... 168 222 1.94%
osd.13 0 75 19 0 211 ... 0 4552 2.06%
osd.20 4026 120 0 197 0 ... 90 0 1.92%
osd.21 120 2181 65 130 116 ... 85 75 1.29%
osd.24 176 150 265 63 0 ... 160 258 2.29%
osd.25 123 99 190 198 99 ... 92 182 2.19%
osd.26 54 83 62 258 254 ... 51 69 2.27%
osd.27 124 109 0 90 73 ... 1840 0 1.55%
osd.29 43 54 0 98 123 ... 1857 0 1.60%
osd.3 74 82 2112 137 153 ... 61 44 1.62%
osd.37 65 108 0 0 166 ... 67 0 1.66%
osd.38 163 119 0 0 73 ... 58 0 1.68%
osd.44 56 73 2250 148 173 ... 77 43 1.68%
osd.46 60 71 132 67 0 ... 39 125 1.31%
osd.47 0 51 70 126 70 ... 0 73 1.35%
osd.8 151 112 163 0 76 ... 0 175 1.67%
osd.9 197 130 202 4493 0 ... 188 0 2.03%
objects% 1.92% 1.29% 1.95% 2.03% 1.89% ... 1.69% 2.06% 33.29%
Continue reading “How many objects will move when changing a crushmap ?”