2021-09-28T22:18:11.765 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-28T22:18:11.767 INFO:teuthology.results:seed: '5539' 2021-09-29T00:47:08.906 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-29T00:47:08.943 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-29T03:13:22.359 INFO:teuthology.results:Tests finished! gathering results... 2021-09-29T03:13:42.882 INFO:scrape:Found 220 jobs 2021-09-29T03:13:58.234 INFO:scrape:Found 9 distinct failure reasons 2021-09-29T03:13:58.235 INFO:scrape:Failure: "2021-09-29T01:03:07.285359+0000 mon.a (mon.0) 144 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2021-09-29T03:13:58.237 INFO:scrape:42 jobs 2021-09-29T03:13:58.238 INFO:scrape:suites intersection: ['conf'] 2021-09-29T03:13:58.239 INFO:scrape:suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/no-object-map', 'features/object-map', 'install/ceph', 'krbd/fsx/{ceph/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{legacy}', 'ms_mode$/{prefer-crc}', 'ms_mode$/{secure}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/few', 'msgr-failures/many', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-1-client}', 'tasks/fsx-3-client}', 'tasks/krbd_data_pool}', 'tasks/krbd_exclusive_option}', 'tasks/krbd_fallocate}', 'tasks/krbd_latest_osdmap_on_map}', 'tasks/krbd_udev_enumerate}', 'tasks/krbd_udev_netlink_enobufs}', 'tasks/rbd_concurrent}', 'tasks/rbd_fio}', 'tasks/rbd_huge_tickets}', 'tasks/rbd_image_read}', 'tasks/rbd_kernel}', 'tasks/rbd_map_snapshot_io}', 'tasks/rbd_map_unmap}', 'tasks/rbd_simple_big}', 'tasks/rbd_workunit_kernel_untar_build}', 'tasks/rbd_workunit_suites_dbench}', 'tasks/rbd_workunit_suites_fsstress_ext4}', 'tasks/rbd_workunit_suites_fsstress}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_suites_iozone}', 'tasks/rbd_workunit_trivial_sync}', 'tasks/rbd_xfstests}'] 2021-09-29T03:13:58.239 INFO:scrape: 2021-09-29T03:13:58.240 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2021-09-29T03:13:58.240 INFO:scrape:1 jobs: ['6412231'] 2021-09-29T03:13:58.240 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2021-09-29T03:13:58.241 INFO:scrape: 2021-09-29T03:13:58.241 INFO:scrape:Failure: Command failed on smithi119 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' 2021-09-29T03:13:58.242 INFO:scrape:4 jobs: ['6412251', '6412385', '6412374', '6412376'] 2021-09-29T03:13:58.242 INFO:scrape:suites intersection: ['conf'] 2021-09-29T03:13:58.243 INFO:scrape:suites union: ['ceph/ceph', 'clusters/3-node', 'clusters/fixed-1', 'clusters/fixed-3', 'conf', 'features/object-map', 'install/ceph', 'krbd/basic/{bluestore-bitmap', 'krbd/fsx/{ceph/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode$/{legacy}', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'randomized-striping-on}', 'striping/fancy/{msgr-failures/few', 'tasks/fsx-1-client}', 'tasks/krbd_data_pool}', 'tasks/krbd_discard}', 'tasks/rbd_concurrent}'] 2021-09-29T03:13:58.243 INFO:scrape: 2021-09-29T03:13:58.244 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/concurrent.sh 2021-09-29T03:13:58.244 INFO:scrape:1 jobs: ['6412287'] 2021-09-29T03:13:58.244 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/crc', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2021-09-29T03:13:58.245 INFO:scrape: 2021-09-29T03:13:58.245 INFO:scrape:Failure: SSH connection to smithi173 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2021-09-29T03:13:58.246 INFO:scrape:1 jobs: ['6412360'] 2021-09-29T03:13:58.247 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/legacy', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2021-09-29T03:13:58.247 INFO:scrape: 2021-09-29T03:13:58.248 INFO:scrape:Failure: SSH connection to smithi058 was lost: 'sudo lsb_release -is' 2021-09-29T03:13:58.248 INFO:scrape:1 jobs: ['6412382'] 2021-09-29T03:13:58.249 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/secure', 'msgr-failures/many', 'tasks/krbd_huge_osdmap}'] 2021-09-29T03:13:58.251 INFO:scrape: 2021-09-29T03:13:58.251 INFO:scrape:Failure: Command failed on smithi098 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude9c2_y7og -r -- '-g auto -g blockdev -x clone'" 2021-09-29T03:13:58.252 INFO:scrape:1 jobs: ['6412229'] 2021-09-29T03:13:58.253 INFO:scrape:suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{prefer-crc}', 'msgr-failures/few', 'tasks/rbd_xfstests}'] 2021-09-29T03:13:58.253 INFO:scrape: 2021-09-29T03:13:58.254 INFO:scrape:Failure: SSH connection to smithi037 was lost: 'rm -f /tmp/linux-image.deb && echo linux-image-5.15.0-rc2-ceph-g252e7632c17c_5.15.0-rc2-ceph-g252e7632c17c-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/wip-rxbounce/252e7632c17cf8c47a8c1197b07ad2edfa8bd851/ubuntu/focal/flavors/default/pool/main/l/linux-5.15.0-rc2-ceph-g252e7632c17c/ --input-file=-' 2021-09-29T03:13:58.254 INFO:scrape:1 jobs: ['6412383'] 2021-09-29T03:13:58.255 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/legacy', 'msgr-failures/few', 'tasks/rbd_workunit_suites_dbench}'] 2021-09-29T03:13:58.255 INFO:scrape: 2021-09-29T03:13:58.256 INFO:scrape:Failure: Command failed on smithi049 with status 151: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 512 -w 512 -t 512 -h 512 -l 250000000 -S 0 -N 20000 -K -U pool_client.0 image_client.0' 2021-09-29T03:13:58.256 INFO:scrape:1 jobs: ['6412357'] 2021-09-29T03:13:58.257 INFO:scrape:suites: ['clusters/3-node', 'conf', 'features/object-map', 'krbd/fsx/{ceph/ceph', 'ms_mode$/{prefer-crc}', 'objectstore/bluestore-bitmap', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-1-client}'] 2021-09-29T03:13:58.257 INFO:scrape: 2021-09-29T03:13:58.266 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi ================================================================= info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/ failed: 49 dead: 4 running: 0 waiting: 0 queued: 0 passed: 167 Fail ================================================================= [6412229] krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:27:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412229/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4281efb094c4a11b05ab6fdee86a132 Command failed on smithi098 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude9c2_y7og -r -- '-g auto -g blockdev -x clone'" [6412231] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412231/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412231/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b67bf1efbef140959bbce58ff9a051e3 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6412233] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_exclusive_option} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412233/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412233/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5043c7b7f97c4d12bd04add4b538f0fb "2021-09-28T22:56:43.972993+0000 mon.a (mon.0) 314 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6412234] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_kernel_untar_build} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412234/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412234/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2455d05445c24bee88c5fb235372d247 "2021-09-28T22:46:47.739239+0000 mon.a (mon.0) 130 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412235] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} ----------------------------------------------------------------- time: 00:19:01 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412235/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412235/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41161d666d6541a8b4fd5637bb431ad8 "2021-09-28T22:57:48.895894+0000 mon.a (mon.0) 278 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log [6412250] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412250/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412250/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7176cf741fd24cab9b5961004a901bb1 "2021-09-28T23:07:18.855185+0000 mon.a (mon.0) 236 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412251] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412251/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412251/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2eacb74e51c454894851585b0ff9e8a Command failed on smithi119 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6412252] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:19:25 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412252/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412252/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c66293ebfdb4947a6cb110fd1a86fbe "2021-09-28T23:07:50.403164+0000 mon.a (mon.0) 293 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412258] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412258/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412258/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69b58dc48300455eb84a9ce6a288590f "2021-09-28T23:06:07.060381+0000 mon.a (mon.0) 129 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log [6412259] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:19:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412259/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412259/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a40c145e0c246c8b361af94b16efa28 "2021-09-28T23:04:20.461832+0000 mon.a (mon.0) 193 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412260] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:18:57 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412260/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412260/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc332fee99e54dfbafb9b139cddf8ead "2021-09-28T23:15:09.404720+0000 mon.a (mon.0) 326 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6412267] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} ----------------------------------------------------------------- time: 00:19:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412267/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412267/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62332034c6764afc9be55eecaca33991 "2021-09-28T23:21:39.101690+0000 mon.a (mon.0) 276 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412270] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_fallocate} ----------------------------------------------------------------- time: 00:19:09 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412270/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6cf75ea315ee49ad892ae2f17d80b0f7 "2021-09-28T23:23:00.595244+0000 mon.a (mon.0) 300 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6412272] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:19:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412272/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412272/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0393d2147e243adb7cfa0a7f3a713e5 "2021-09-28T23:23:46.413263+0000 mon.a (mon.0) 322 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412275] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:18:15 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412275/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c391a60acdf34aeca8df5792837a2635 "2021-09-28T23:25:51.432705+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 5 osds down (OSD_DOWN)" in cluster log [6412278] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412278/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412278/ sentry: https://sentry.ceph.com/organizations/ceph/?query=396c1b39241547108e90b6c72b1a9388 "2021-09-28T23:18:06.881691+0000 mon.a (mon.0) 135 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412280] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:19:23 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412280/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64f5a1becdc6477fb08973bf5474dfe6 "2021-09-28T23:18:36.843456+0000 mon.a (mon.0) 159 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412287] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 03:59:44 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412287/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=634786acf0404d2d9e130d2c3b9852dc Command failed (workunit test rbd/concurrent.sh) on smithi142 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=84379239c065eab5d8e5ce78b9ef546a60428aff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh' [6412290] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:20:05 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412290/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20b8e0e54340453583774e5bd419fa5f "2021-09-28T23:32:57.756905+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412295] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:19:46 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412295/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae885f1a00f34f3583deed5e9edb5b16 "2021-09-28T23:30:21.876810+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412297] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:19:18 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412297/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412297/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65e1b3fbc29a4fa3b857431b2805a8c8 "2021-09-28T23:30:52.993180+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412298] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_simple_big} ----------------------------------------------------------------- time: 00:19:15 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412298/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412298/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1551bae99fa949358969dcba16ee7cb6 "2021-09-28T23:40:56.948667+0000 mon.a (mon.0) 229 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412317] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412317/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dee20f6ceb6042799470c347f2b96d3f "2021-09-28T23:54:29.376034+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412325] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:19:31 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412325/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412325/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00b175be55fe4bf0a1efd7a7bc910364 "2021-09-28T23:58:13.751421+0000 mon.a (mon.0) 332 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412329] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412329/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412329/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5a77a4417e3449c92e20aa47b05417c "2021-09-28T23:48:44.760277+0000 mon.a (mon.0) 138 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412330] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412330/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412330/ sentry: https://sentry.ceph.com/organizations/ceph/?query=180b8f02ce9f45fd8b888871f55ca374 "2021-09-28T23:59:02.640039+0000 mon.a (mon.0) 276 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412333] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_unmap} ----------------------------------------------------------------- time: 00:19:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412333/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9163e77c1039436b920705e8b5771939 "2021-09-29T00:00:54.978961+0000 mon.a (mon.0) 244 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412340] krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 00:19:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412340/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5575370346474e8ca3b1c52cd981e4ab "2021-09-29T00:02:02.932305+0000 mon.a (mon.0) 91 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412344] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} ----------------------------------------------------------------- time: 00:19:49 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412344/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3211e8e00be47feb4306fa791abc2f7 "2021-09-29T00:12:49.853008+0000 mon.a (mon.0) 315 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412347] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:19:20 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412347/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5f89337142a44ed9bbeac09bcefc83a "2021-09-29T00:13:13.481377+0000 mon.a (mon.0) 270 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412354] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412354/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=553172e903d54b409861a24b96f06578 "2021-09-29T00:10:14.201796+0000 mon.a (mon.0) 114 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412357] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:12:42 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412357/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fe7d012ca41412986cb2f8ee6073efc Command failed on smithi049 with status 151: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 512 -w 512 -t 512 -h 512 -l 250000000 -S 0 -N 20000 -K -U pool_client.0 image_client.0' [6412362] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:37 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412362/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f372fc2de04418dbc370031df972d2a "2021-09-29T00:24:15.003348+0000 mon.a (mon.0) 229 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412363] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:19:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412363/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1375e77c47e9452a974f7b78c9ac08a4 "2021-09-29T00:24:44.985770+0000 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412369] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412369/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412369/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7963a48d4c9d4372a0485b0c7404f32e "2021-09-29T00:28:53.234878+0000 mon.a (mon.0) 349 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412376] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412376/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412376/ sentry: https://sentry.ceph.com/organizations/ceph/?query=effbb052607b42c08c11e2e8576893fd Command failed on smithi037 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6412384] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412384/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412384/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f84ae5365074ae6b36ebd0bc7f4bc9d "2021-09-29T00:30:14.034501+0000 mon.a (mon.0) 194 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412385] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412385/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3c0cad8ab8b482681636398e4ed11aa Command failed on smithi037 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6412389] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} ----------------------------------------------------------------- time: 00:19:11 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412389/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3521f5f649548d48b5ce0c87611590a "2021-09-29T00:33:53.801687+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412390] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:42 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412390/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b581957ea05042e6af06a354e7b10fc4 "2021-09-29T00:35:28.781863+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412405] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412405/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f2de8535f3d4efb8dcb0bfec5c24378 "2021-09-29T00:52:12.622019+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412411] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:18:21 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412411/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe866f26073d4877b955ac27661f929b "2021-09-29T00:45:25.834427+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412412] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_fio} ----------------------------------------------------------------- time: 00:19:12 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412412/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412412/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69b42a418116491b827445be87a7b02d "2021-09-29T00:56:30.729737+0000 mon.a (mon.0) 299 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412413] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_data_pool} ----------------------------------------------------------------- time: 00:19:38 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412413/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=780ef44e11984354971881ead9c591b7 "2021-09-29T00:56:37.684979+0000 mon.a (mon.0) 281 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412426] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412426/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d157925e09b3429ba2cd279e2970c010 "2021-09-29T01:11:44.178149+0000 mon.a (mon.0) 291 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412433] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412433/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412433/ sentry: https://sentry.ceph.com/organizations/ceph/?query=657602bf6bc940e5bdd9279734bb8e87 "2021-09-29T01:12:26.763789+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6412435] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:20:18 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412435/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412435/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b1d1c5340544a2fa9e827af0d2f7c2a "2021-09-29T01:03:07.285359+0000 mon.a (mon.0) 144 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6412438] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:19:36 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412438/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54506913b2cf4b85982760e014968f81 "2021-09-29T01:14:19.638482+0000 mon.a (mon.0) 296 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6412441] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412441/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412441/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2cadcbd284d453dba19af91161bcbc4 "2021-09-29T01:15:33.600497+0000 mon.a (mon.0) 227 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log Dead ================================================================= [6412360] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 02:49:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412360/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e1b5a8278d74b958e7637b208d83d05 SSH connection to smithi173 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6412374] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412374/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9977a2df2fab4f32bcfbc10790d48159 SSH connection to smithi058 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6412382] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412382/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dccffb9c6e9d4ddeb721c3e8dcefd321 SSH connection to smithi058 was lost: 'sudo lsb_release -is' [6412383] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412383/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d1d8aa2b1214587b5ed18886e8c8ec8 SSH connection to smithi037 was lost: 'rm -f /tmp/linux-image.deb && echo linux-image-5.15.0-rc2-ceph-g252e7632c17c_5.15.0-rc2-ceph-g252e7632c17c-1_a md64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/wip-rxbounce/252e7632c17cf8c47a8c 1197b07ad2edfa8bd851/ubuntu/focal/flavors/default/pool/main/l/linux-5.15.0- rc2-ceph-g252e7632c17c/ --input-file=-' Pass ================================================================= [6412225] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_deep_flatten} time: 00:09:15 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412225/ [6412226] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:20:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412226/ [6412227] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_fio} time: 00:28:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412227/ [6412228] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_data_pool} time: 00:17:58 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412228/ [6412230] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_fio} time: 00:43:58 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412230/ [6412232] krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} time: 00:08:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412232/ [6412236] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_huge_osdmap} time: 00:11:37 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412236/ [6412237] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412237/ [6412238] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:13:56 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412238/ [6412239] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_discard} time: 00:09:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412239/ [6412240] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:17:26 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412240/ [6412241] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} time: 00:11:43 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412241/ [6412242] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:25:52 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412242/ [6412243] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_enumerate} time: 00:35:25 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412243/ [6412244] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netlink_enobufs} time: 00:17:43 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412244/ [6412245] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:12:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412245/ [6412246] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} time: 00:12:11 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412246/ [6412247] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} time: 00:08:52 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412247/ [6412248] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:23:59 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412248/ [6412249] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_symlinks} time: 00:11:58 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412249/ [6412253] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:18:40 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412253/ [6412254] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_image_read} time: 00:10:40 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412254/ [6412255] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_modprobe} time: 00:08:55 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412255/ [6412256] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:15:36 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412256/ [6412257] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:17:16 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412257/ [6412261] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_unmap} time: 00:15:18 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412261/ [6412262] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412262/ [6412263] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_simple_big} time: 00:23:58 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412263/ [6412264] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_msgr_segments} time: 00:08:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412264/ [6412265] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:24:49 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412265/ [6412266] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} time: 00:28:42 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412266/ [6412268] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412268/ [6412269] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:56 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412269/ [6412271] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} time: 00:11:23 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412271/ [6412273] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:14:33 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412273/ [6412274] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} time: 00:09:03 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412274/ [6412276] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_namespaces} time: 00:11:59 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412276/ [6412277] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:15:12 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412277/ [6412279] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:16:53 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412279/ [6412281] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netns} time: 00:11:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412281/ [6412282] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_read_only} time: 00:09:15 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412282/ [6412283] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:18:04 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412283/ [6412284] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/pggrow thrashosds-health workloads/rbd_fio} time: 00:37:31 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412284/ [6412285] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} time: 00:12:38 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412285/ [6412286] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412286/ [6412288] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_huge_tickets} time: 00:11:45 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412288/ [6412289] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:21:26 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412289/ [6412291] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_whole_object_zeroout} time: 00:09:09 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412291/ [6412292] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:18:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412292/ [6412293] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_kernel} time: 00:10:08 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412293/ [6412294] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:03 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412294/ [6412296] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} time: 00:15:48 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412296/ [6412299] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} time: 00:09:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412299/ [6412300] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:22:55 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412300/ [6412301] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_fio} time: 00:29:11 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412301/ [6412302] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_data_pool} time: 00:18:04 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412302/ [6412303] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} time: 00:09:35 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412303/ [6412304] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} time: 00:20:48 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412304/ [6412305] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:37 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412305/ [6412306] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:39 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412306/ [6412307] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_fallocate} time: 00:11:23 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412307/ [6412308] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_huge_osdmap} time: 00:11:49 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412308/ [6412309] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:38 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412309/ [6412310] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:13:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412310/ [6412311] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_discard} time: 00:09:33 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412311/ [6412312] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:17:27 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412312/ [6412313] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:18:36 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412313/ [6412314] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_namespaces} time: 00:12:01 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412314/ [6412315] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:20:59 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412315/ [6412316] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_enumerate} time: 00:35:07 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412316/ [6412318] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:11:57 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412318/ [6412319] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netns} time: 00:11:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412319/ [6412320] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_huge_image} time: 00:08:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412320/ [6412321] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:23:40 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412321/ [6412322] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_symlinks} time: 00:12:10 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412322/ [6412323] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412323/ [6412324] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} time: 00:15:49 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412324/ [6412326] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:18:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412326/ [6412327] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_image_read} time: 00:11:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412327/ [6412328] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} time: 00:08:45 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412328/ [6412331] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:17:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412331/ [6412332] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:11 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412332/ [6412334] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:33 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412334/ [6412335] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} time: 00:25:15 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412335/ [6412336] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_msgr_segments} time: 00:08:56 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412336/ [6412337] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:23:31 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412337/ [6412338] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_fio} time: 00:28:42 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412338/ [6412339] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_data_pool} time: 00:22:12 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412339/ [6412341] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:22:39 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412341/ [6412342] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412342/ [6412343] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412343/ [6412345] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_huge_osdmap} time: 00:12:11 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412345/ [6412346] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} time: 00:41:01 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412346/ [6412348] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_parent_overlap} time: 00:09:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412348/ [6412349] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:20:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412349/ [6412350] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} time: 00:11:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412350/ [6412351] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:18:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412351/ [6412352] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_enumerate} time: 00:38:27 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412352/ [6412353] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:16:30 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412353/ [6412355] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} time: 00:11:27 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412355/ [6412356] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} time: 00:09:28 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412356/ [6412358] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_symlinks} time: 00:11:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412358/ [6412359] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:10:56 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412359/ [6412361] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} time: 00:11:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412361/ [6412364] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_whole_object_zeroout} time: 00:09:10 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412364/ [6412365] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:14:28 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412365/ [6412366] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} time: 00:37:07 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412366/ [6412367] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_kernel} time: 00:10:23 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412367/ [6412368] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412368/ [6412370] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_unmap} time: 00:15:17 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412370/ [6412371] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:12:16 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412371/ [6412372] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_simple_big} time: 00:25:57 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412372/ [6412373] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_deep_flatten} time: 00:09:29 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412373/ [6412375] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} time: 00:28:08 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412375/ [6412377] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} time: 00:09:54 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412377/ [6412378] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} time: 00:20:52 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412378/ [6412379] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:39 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412379/ [6412380] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:24 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412380/ [6412381] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_fallocate} time: 00:11:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412381/ [6412386] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:16:23 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412386/ [6412387] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_namespaces} time: 00:12:05 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412387/ [6412388] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:22:26 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412388/ [6412391] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:12:07 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412391/ [6412392] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netns} time: 00:11:52 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412392/ [6412393] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_huge_image} time: 00:09:06 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412393/ [6412394] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:21:07 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412394/ [6412395] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:09:28 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412395/ [6412396] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} time: 00:12:41 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412396/ [6412397] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:48 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412397/ [6412398] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_concurrent} time: 00:16:04 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412398/ [6412399] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_huge_tickets} time: 00:10:45 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412399/ [6412400] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:19:10 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412400/ [6412401] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} time: 00:11:02 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412401/ [6412402] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_modprobe} time: 00:08:44 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412402/ [6412403] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:18:05 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412403/ [6412404] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_kernel} time: 00:10:37 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412404/ [6412406] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:17 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412406/ [6412407] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} time: 00:16:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412407/ [6412408] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:50 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412408/ [6412409] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_simple_big} time: 00:25:16 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412409/ [6412410] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} time: 00:09:13 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412410/ [6412414] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:22 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412414/ [6412415] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:59 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412415/ [6412416] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_fallocate} time: 00:11:33 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412416/ [6412417] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_huge_osdmap} time: 00:11:54 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412417/ [6412418] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_dbench} time: 00:39:13 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412418/ [6412419] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:14:04 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412419/ [6412420] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_parent_overlap} time: 00:09:18 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412420/ [6412421] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:15:16 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412421/ [6412422] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_fio} time: 00:40:20 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412422/ [6412423] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_namespaces} time: 00:11:23 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412423/ [6412424] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:16:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412424/ [6412425] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_enumerate} time: 00:39:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412425/ [6412427] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:13:32 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412427/ [6412428] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netns} time: 00:12:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412428/ [6412429] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_read_only} time: 00:09:04 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412429/ [6412430] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:24:31 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412430/ [6412431] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_symlinks} time: 00:12:48 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412431/ [6412432] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:01 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412432/ [6412434] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_huge_tickets} time: 00:11:51 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412434/ [6412436] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_image_read} time: 00:11:36 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412436/ [6412437] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} time: 00:09:46 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412437/ [6412439] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} time: 00:11:01 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412439/ [6412440] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:19 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412440/ [6412442] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_unmap} time: 00:15:25 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412442/ [6412443] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:12:34 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412443/ [6412444] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} time: 00:27:44 info: http://pulpito.ceph.com/dis-2021-09-28_22:17:57-krbd-master-wip-rxbounce-basic-smithi/6412444/