When investingating a a problem using the latest Ceph sources, it was discovered that the problem only shows in the master branch and appeared after the v0.85 tag. The following script reproduces the problem and logs the result:
$ cat try.sh #!/bin/bash cd src log=$(git describe) echo $log.log make -j4 >& $log.log rm -fr dev out ; mkdir -p dev MDS=1 MON=1 OSD=3 timeout 120 ./vstart.sh \ -o 'paxos propose interval = 0.01' \ -n -l mon osd mds >> $log.log 2>&1 status=$? ./stop.sh exit $status
It can be used with git bisect to find the revision in which it first appeared.
$ git bisect start # initialize the search $ git bisect bad origin/master # the problem happens $ git bisect good tags/v0.85 # the problem does not happen $ git bisect skip $(git log --format='%H' --no-merges tags/v0.85..origin/master) $ git bisect run try.sh # binary search in tags/v0.85..origin/master running try.sh v0.85-679-g8d3f135.log Bisecting: 339 revisions left to test after this (roughly 8 steps) [ef006ae] Merge pull request #2658 from athanatos/wip-9625 running try.sh v0.86-27-gef006ae.log Bisecting: 169 revisions left to test after this (roughly 7 steps) [fa0bd06] ceph-disk: bootstrap-osd keyring ignores --statedir running try.sh v0.85-1116-gfa0bd06.log ... v0.86-263-g5f6589c.log d15ecafea4 is the first bad commit commit d15eca Author: John Spray
Date: Fri Sep 26 17:24:12 2014 +0100 vstart: create fewer pgs for fs pools :040000 040000 f42a324a8 aa64cdc1ed3 M src bisect run success
The git bisect skip excludes all non merge commits from the search. The branches are carefully tested before being merged and are, at least, known to pass make check successfully. The individual commits within a branch are unlikely to pass make check and some of them may not even compile.
The information displayed by git bisect run is terse when it ends with skipped commits:
Bisecting: 39 revisions left to test after this (roughly 5 steps) [083c2e42c663229ce505f74c40d8261ca530a79b] Merge pull request #6565 from chenji-kael/patch-1 running /home/loic/ceph-centos-7-loic/try.sh v9.2.0-920-g083c2e4.log There are only 'skip'ped commits left to test. The first bad commit could be any of: 536c70281a8952358e8d88a6ff8d7cd9b8db5a76
To get more detailed, git bisect log can be used:
# good: [b584388ce9ce998c99e219ec144725beaf09ab28] Merge pull request #6489 from xiexingguo/xxg-wip-13715 git bisect good b584388ce9ce998c99e219ec144725beaf09ab28 # bad: [5135292d9557269bab5cefc98d39606174aa6ebe] Merge branch 'wip-bigbang' git bisect bad 5135292d9557269bab5cefc98d39606174aa6ebe # good: [f3e88ace74c896c72f6e8485c44c7432f298d887] Merge remote-tracking branch 'gh/jewel' git bisect good f3e88ace74c896c72f6e8485c44c7432f298d887 # good: [083c2e42c663229ce505f74c40d8261ca530a79b] Merge pull request #6565 from chenji-kael/patch-1 git bisect good 083c2e42c663229ce505f74c40d8261ca530a79b # only skipped commits left to test # possible first bad commit: [5135292d9557269bab5cefc98d39606174aa6ebe] Merge branch 'wip-bigbang' # possible first bad commit: [9aabc8a9b8d7775337716c4e0fa3cc53938acb45] test/mon/osd-crush.sh: escape ceph tell mon.* # possible first bad commit: [72edab282343e8509b387f92d05fc4d6ae96b25b] osd: make some of the pg_temp methods/fields private # possible first bad commit: [987f68a8df292668ad241f4769d82792644454dd] osdc/Objecter: call notify completion only once # possible first bad commit: [d201c6d93f40affe72d940605c8786247451d3e5] mon: change mon_osd_min_down_reporters from 1 -> 2