2022-02-24T20:32:16.991 INFO:root:teuthology version: 1.1.0-5825e8d0 2022-02-24T20:32:16.992 INFO:teuthology.results:seed: '3770' 2022-02-25T04:06:01.323 INFO:root:teuthology version: 1.1.0-5825e8d0 2022-02-25T04:06:01.365 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-25T07:46:19.988 INFO:teuthology.results:Tests finished! gathering results... 2022-02-25T07:46:40.880 INFO:scrape:Found 221 jobs 2022-02-25T07:46:51.007 INFO:scrape:Found 5 distinct failure reasons 2022-02-25T07:46:51.009 INFO:scrape:Failure: "2022-02-25T03:13:07.060664+0000 mon.a (mon.0) 182 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2022-02-25T07:46:51.009 INFO:scrape:28 jobs: ['6693506', '6693474', '6693438', '6693446', '6693470', '6693489', '6693408', '6693593', '6693459', '6693483', '6693432', '6693414', '6693488', '6693500', '6693513', '6693461', '6693556', '6693456', '6693582', '6693415', '6693523', '6693439', '6693607', '6693585', '6693609', '6693601', '6693495', '6693570'] 2022-02-25T07:46:51.010 INFO:scrape:suites intersection: ['conf'] 2022-02-25T07:46:51.011 INFO:scrape:suites union: ['ceph/ceph', 'clusters/3-node', 'clusters/fixed-3', 'conf', 'features/no-object-map', 'features/object-map', 'install/ceph', 'krbd/fsx/{ceph/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'krbd/singleton/{bluestore-bitmap', 'krbd/wac/wac/{bluestore-bitmap', 'ms_mode$/{crc-rxbounce}', 'ms_mode$/{secure}', 'ms_mode/crc$/{crc-rxbounce}', 'ms_mode/crc$/{crc}', 'ms_mode/legacy$/{legacy-rxbounce}', 'ms_mode/legacy$/{legacy}', 'ms_mode/secure', 'msgr-failures/few', 'msgr-failures/many', 'objectstore/bluestore-bitmap', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-1-client}', 'tasks/fsx-3-client}', 'tasks/krbd_fallocate}', 'tasks/krbd_huge_osdmap}', 'tasks/krbd_latest_osdmap_on_map}', 'tasks/krbd_udev_netlink_enobufs}', 'tasks/krbd_udev_symlinks}', 'tasks/rbd_fio}', 'tasks/rbd_image_read}', 'tasks/rbd_kernel}', 'tasks/rbd_map_snapshot_io}', 'tasks/rbd_workunit_suites_dbench}', 'tasks/rbd_workunit_suites_ffsb}', 'tasks/rbd_workunit_suites_fsstress_ext4}', 'tasks/rbd_workunit_suites_fsstress}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_trivial_sync}', 'tasks/rbd_xfstests}', 'tasks/wac', 'verify/many-resets}'] 2022-02-25T07:46:51.011 INFO:scrape: 2022-02-25T07:46:51.012 INFO:scrape:Failure: Command failed on smithi188 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude6nc1n8kn -r -- '-g auto -g blockdev -x clone'" 2022-02-25T07:46:51.012 INFO:scrape:1 jobs: ['6693395'] 2022-02-25T07:46:51.013 INFO:scrape:suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{legacy}', 'msgr-failures/few', 'tasks/rbd_xfstests}'] 2022-02-25T07:46:51.013 INFO:scrape: 2022-02-25T07:46:51.014 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/concurrent.sh 2022-02-25T07:46:51.014 INFO:scrape:1 jobs: ['6693599'] 2022-02-25T07:46:51.014 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/secure', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2022-02-25T07:46:51.015 INFO:scrape: 2022-02-25T07:46:51.015 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2022-02-25T07:46:51.016 INFO:scrape:1 jobs: ['6693397'] 2022-02-25T07:46:51.016 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2022-02-25T07:46:51.017 INFO:scrape: 2022-02-25T07:46:51.017 INFO:scrape:Failure: SSH connection to smithi184 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-02-25T07:46:51.018 INFO:scrape:1 jobs: ['6693526'] 2022-02-25T07:46:51.018 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/legacy$/{legacy}', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2022-02-25T07:46:51.018 INFO:scrape: 2022-02-25T07:46:51.026 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/ failed: 31 dead: 1 running: 0 waiting: 0 queued: 0 passed: 189 Fail ================================================================= [6693395] krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:27:47 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693395/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f55109ee46e426598a5d8d77358a77e Command failed on smithi188 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude6nc1n8kn -r -- '-g auto -g blockdev -x clone'" [6693397] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693397/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5d06898e22e4bdba7899da519d4a125 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6693408] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:19:52 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693408/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8030c95db76546d5bb93c3654ff3a0c2 "2022-02-24T21:17:09.161434+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693414] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:19:31 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693414/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a331a26c0904e8084a7dbfda6ff228f "2022-02-24T21:31:43.918269+0000 mon.a (mon.0) 293 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693415] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} ----------------------------------------------------------------- time: 00:19:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693415/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83813c79a324448099ddc0b6e2b77837 "2022-02-24T21:32:15.117199+0000 mon.a (mon.0) 300 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693432] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} ----------------------------------------------------------------- time: 00:19:00 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693432/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693432/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e51122010d6848efb781d74684048287 "2022-02-24T21:42:20.050511+0000 mon.a (mon.0) 258 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693438] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:19:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693438/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e00ec52e4dde48cebd0eaf3ca75b38a3 "2022-02-24T21:47:29.614945+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693439] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693439/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693439/ sentry: https://sentry.ceph.com/organizations/ceph/?query=204eb6ca64ba4abb92c877d0905a64ff "2022-02-25T02:16:40.031859+0000 mon.a (mon.0) 213 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693446] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:19:47 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693446/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693446/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e4d48cf6cd04a09b304507bbe80178a "2022-02-25T02:22:48.146972+0000 mon.a (mon.0) 307 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693456] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:19:15 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693456/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693456/ sentry: https://sentry.ceph.com/organizations/ceph/?query=add5f2366b9047de986e4c840e05e113 "2022-02-25T02:30:43.205312+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693459] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:10 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693459/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b163b9f19732492a9ef061d746d949df "2022-02-25T02:23:06.396253+0000 mon.a (mon.0) 131 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693461] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:19:38 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693461/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5c3fa818bdc4ba98bfa16377564e484 "2022-02-25T02:36:10.939988+0000 mon.a (mon.0) 291 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693470] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} ----------------------------------------------------------------- time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693470/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51a0cf9928684ffc80c8c0ca94275d80 "2022-02-25T02:31:44.496695+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693474] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693474/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b127af1482e4246a352d576b1ba4e56 "2022-02-25T02:36:37.339203+0000 mon.a (mon.0) 193 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693483] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:27 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693483/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=364fcfc32dac4d15a7ca82197690c748 "2022-02-25T02:45:17.561929+0000 mon.a (mon.0) 161 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693488] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_symlinks} ----------------------------------------------------------------- time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693488/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693488/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea9694fff502468499e6dd07781bb9cb "2022-02-25T03:01:51.123831+0000 mon.a (mon.0) 215 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693489] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} ----------------------------------------------------------------- time: 00:20:09 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693489/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693489/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79bc15f0527d4a978ce14d2eaa284211 "2022-02-25T03:03:28.048007+0000 mon.a (mon.0) 301 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693495] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:19:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693495/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693495/ sentry: https://sentry.ceph.com/organizations/ceph/?query=988e7de8e55943759812ef67e32e23e0 "2022-02-25T03:04:09.359972+0000 mon.a (mon.0) 281 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693500] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:19:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693500/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6a6b0667a244302895cc19afa8ac30a "2022-02-25T02:59:13.548806+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693506] krbd/singleton/{bluestore-bitmap conf ms_mode$/{secure} msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 00:18:56 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693506/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8cef88e5478453295b54cf690646f34 "2022-02-25T03:13:07.060664+0000 mon.a (mon.0) 182 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693513] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:19:25 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693513/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693513/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1aab30c2ff44cb998a0ece14ec4ab26 "2022-02-25T03:11:43.581961+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693523] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:18:59 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693523/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1c3ffaa696742cea90ad0db13582572 "2022-02-25T03:28:36.763707+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693556] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693556/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693556/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b696d131f9784cf69a60a1ae05744520 "2022-02-25T03:52:55.058059+0000 mon.a (mon.0) 258 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693570] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:36 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693570/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693570/ sentry: https://sentry.ceph.com/organizations/ceph/?query=331b39f7955340f3a3b32a4c917102f2 "2022-02-25T04:04:14.610270+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693582] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} ----------------------------------------------------------------- time: 00:19:10 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693582/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e88c4db28ab74eeabbefdc5e2b91c499 "2022-02-25T04:13:53.273184+0000 mon.a (mon.0) 253 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693585] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:19:11 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693585/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693585/ sentry: https://sentry.ceph.com/organizations/ceph/?query=639fc0248e474fdda8f970692def1765 "2022-02-25T04:05:42.776213+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6693593] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:14:42 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693593/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693593/ "2022-02-25T04:10:46.618373+0000 osd.4 (osd.4) 3 : cluster [WRN] 1 slow requests (by type [ 'started' : 1 ] most affected pool [ 'rbd' : 1 ])" in cluster log [6693599] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 03:39:15 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693599/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693599/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3181ece7e8f44af8da4f3769b42c7e2 Command failed (workunit test rbd/concurrent.sh) on smithi165 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cee3cae9b2d1ea14db0ffaf3984486d278ff8871 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh' [6693601] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:48 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693601/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693601/ sentry: https://sentry.ceph.com/organizations/ceph/?query=574098ed81dd475db6eae3a9f768f7f8 "2022-02-25T04:26:13.608643+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693607] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:19:25 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693607/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693607/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09bfb8f426044cff85760cc8ea585a63 "2022-02-25T04:29:57.758813+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6693609] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:19:04 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693609/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693609/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4cb48ef9cb0441859813f1b63cca219d "2022-02-25T04:30:45.553390+0000 mon.a (mon.0) 258 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log Dead ================================================================= [6693526] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 03:02:06 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693526/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693526/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0bb89c4167e4f30ada00f1c725305ff SSH connection to smithi184 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' Pass ================================================================= [6693390] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_deep_flatten} time: 00:09:27 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693390/ [6693391] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:20:00 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693391/ [6693392] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_fio} time: 00:28:36 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693392/ [6693393] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_data_pool} time: 00:18:46 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693393/ [6693394] krbd/rxbounce/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce} time: 00:11:36 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693394/ [6693396] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/backoff thrashosds-health workloads/rbd_fio} time: 00:38:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693396/ [6693398] krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} time: 00:09:33 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693398/ [6693399] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:41 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693399/ [6693400] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:41 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693400/ [6693401] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} time: 00:11:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693401/ [6693402] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} time: 00:11:35 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693402/ [6693403] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:00 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693403/ [6693404] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:13:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693404/ [6693405] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_discard} time: 00:09:20 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693405/ [6693406] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:17:45 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693406/ [6693407] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} time: 00:11:12 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693407/ [6693409] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_enumerate} time: 00:37:10 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693409/ [6693410] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netlink_enobufs} time: 00:14:59 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693410/ [6693411] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:11:45 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693411/ [6693412] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} time: 00:11:05 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693412/ [6693413] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} time: 00:09:08 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693413/ [6693416] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693416/ [6693417] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_concurrent} time: 00:16:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693417/ [6693418] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} time: 00:10:42 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693418/ [6693419] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:18:46 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693419/ [6693420] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_image_read} time: 00:10:20 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693420/ [6693421] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_modprobe} time: 00:09:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693421/ [6693422] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:15:32 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693422/ [6693423] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:13:22 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693423/ [6693424] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_kernel} time: 00:10:39 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693424/ [6693425] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:17:37 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693425/ [6693426] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:13 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693426/ [6693427] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_unmap} time: 00:15:13 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693427/ [6693428] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:26 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693428/ [6693429] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_simple_big} time: 00:22:39 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693429/ [6693430] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_msgr_segments} time: 00:09:29 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693430/ [6693431] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:21:44 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693431/ [6693433] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} time: 00:23:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693433/ [6693434] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:35 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693434/ [6693435] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:25 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693435/ [6693436] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_fallocate} time: 00:11:49 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693436/ [6693437] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} time: 00:11:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693437/ [6693440] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} time: 00:09:28 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693440/ [6693441] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:18:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693441/ [6693442] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_namespaces} time: 00:11:36 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693442/ [6693443] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:14:51 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693443/ [6693444] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} time: 00:39:37 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693444/ [6693445] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:15:05 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693445/ [6693447] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netns} time: 00:11:31 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693447/ [6693448] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_read_only} time: 00:09:41 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693448/ [6693449] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:20:46 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693449/ [6693450] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/pggrow thrashosds-health workloads/rbd_fio} time: 00:38:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693450/ [6693451] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} time: 00:11:22 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693451/ [6693452] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:10:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693452/ [6693453] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} time: 00:15:47 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693453/ [6693454] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_huge_tickets} time: 00:10:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693454/ [6693455] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:21:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693455/ [6693457] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_whole_object_zeroout} time: 00:09:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693457/ [6693458] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:18:42 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693458/ [6693460] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693460/ [6693462] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} time: 00:15:19 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693462/ [6693463] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:12:20 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693463/ [6693464] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_simple_big} time: 00:24:08 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693464/ [6693465] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} time: 00:09:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693465/ [6693466] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:22:49 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693466/ [6693467] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_fio} time: 00:28:08 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693467/ [6693468] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_data_pool} time: 00:18:40 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693468/ [6693469] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} time: 00:09:29 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693469/ [6693471] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:29 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693471/ [6693472] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:10 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693472/ [6693473] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_fallocate} time: 00:11:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693473/ [6693475] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:46 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693475/ [6693476] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:13:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693476/ [6693477] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_discard} time: 00:09:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693477/ [6693478] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:18:09 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693478/ [6693479] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:18:08 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693479/ [6693480] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_namespaces} time: 00:11:47 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693480/ [6693481] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:19:52 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693481/ [6693482] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_enumerate} time: 00:36:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693482/ [6693484] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693484/ [6693485] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netns} time: 00:11:01 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693485/ [6693486] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_huge_image} time: 00:09:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693486/ [6693487] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:22:36 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693487/ [6693490] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} time: 00:15:49 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693490/ [6693491] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_huge_tickets} time: 00:11:13 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693491/ [6693492] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:19:18 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693492/ [6693493] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_image_read} time: 00:10:50 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693493/ [6693494] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} time: 00:08:59 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693494/ [6693496] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} time: 00:10:40 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693496/ [6693497] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:18:07 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693497/ [6693498] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693498/ [6693499] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_unmap} time: 00:15:41 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693499/ [6693501] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} time: 00:24:54 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693501/ [6693502] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_msgr_segments} time: 00:09:25 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693502/ [6693503] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:24:15 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693503/ [6693504] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio} time: 00:28:30 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693504/ [6693505] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_data_pool} time: 00:22:21 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693505/ [6693507] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:21:33 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693507/ [6693508] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:49 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693508/ [6693509] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:22 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693509/ [6693510] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} time: 00:11:42 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693510/ [6693511] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_huge_osdmap} time: 00:11:22 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693511/ [6693512] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} time: 00:37:22 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693512/ [6693514] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_parent_overlap} time: 00:08:50 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693514/ [6693515] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:17:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693515/ [6693516] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693516/ [6693517] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:14:54 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693517/ [6693518] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_enumerate} time: 00:39:34 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693518/ [6693519] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:15:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693519/ [6693520] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:13:06 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693520/ [6693521] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} time: 00:11:20 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693521/ [6693522] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} time: 00:09:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693522/ [6693524] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_symlinks} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693524/ [6693525] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693525/ [6693527] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693527/ [6693528] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:20:45 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693528/ [6693529] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_image_read} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693529/ [6693530] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_whole_object_zeroout} time: 00:09:21 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693530/ [6693531] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:14:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693531/ [6693532] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} time: 00:38:25 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693532/ [6693533] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_kernel} time: 00:10:18 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693533/ [6693534] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:00 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693534/ [6693535] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:10:38 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693535/ [6693536] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_unmap} time: 00:15:57 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693536/ [6693537] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:13:05 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693537/ [6693538] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_simple_big} time: 00:23:40 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693538/ [6693539] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_deep_flatten} time: 00:09:55 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693539/ [6693540] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:23:06 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693540/ [6693541] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} time: 00:28:01 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693541/ [6693542] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693542/ [6693543] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} time: 00:09:15 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693543/ [6693544] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} time: 00:21:11 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693544/ [6693545] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693545/ [6693546] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:33 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693546/ [6693547] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_fallocate} time: 00:11:08 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693547/ [6693548] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693548/ [6693549] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693549/ [6693550] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:13:33 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693550/ [6693551] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} time: 00:09:51 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693551/ [6693552] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:16:44 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693552/ [6693553] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_namespaces} time: 00:11:50 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693553/ [6693554] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:21:28 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693554/ [6693555] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} time: 00:36:49 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693555/ [6693557] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:12:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693557/ [6693558] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netns} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693558/ [6693559] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_huge_image} time: 00:09:35 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693559/ [6693560] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:22:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693560/ [6693561] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:18:55 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693561/ [6693562] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} time: 00:11:57 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693562/ [6693563] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:10:43 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693563/ [6693564] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_concurrent} time: 00:15:35 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693564/ [6693565] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_huge_tickets} time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693565/ [6693566] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:19:07 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693566/ [6693567] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693567/ [6693568] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_modprobe} time: 00:08:47 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693568/ [6693569] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:19:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693569/ [6693571] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:17:00 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693571/ [6693572] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693572/ [6693573] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} time: 00:15:04 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693573/ [6693574] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:29 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693574/ [6693575] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_simple_big} time: 00:22:33 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693575/ [6693576] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} time: 00:08:45 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693576/ [6693577] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:25:48 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693577/ [6693578] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_fio} time: 00:28:26 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693578/ [6693579] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_data_pool} time: 00:22:57 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693579/ [6693580] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:39 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693580/ [6693581] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693581/ [6693583] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_huge_osdmap} time: 00:11:27 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693583/ [6693584] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_dbench} time: 00:37:49 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693584/ [6693586] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_parent_overlap} time: 00:09:24 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693586/ [6693587] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:15:13 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693587/ [6693588] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_fio} time: 00:39:53 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693588/ [6693589] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_namespaces} time: 00:11:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693589/ [6693590] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:13:35 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693590/ [6693591] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_enumerate} time: 00:42:47 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693591/ [6693592] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:15:13 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693592/ [6693594] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netns} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693594/ [6693595] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_read_only} time: 00:10:04 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693595/ [6693596] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:22:26 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693596/ [6693597] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693597/ [6693598] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:10:32 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693598/ [6693600] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_huge_tickets} time: 00:10:17 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693600/ [6693602] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_image_read} time: 00:11:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693602/ [6693603] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} time: 00:09:37 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693603/ [6693604] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:17:16 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693604/ [6693605] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} time: 00:10:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693605/ [6693606] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:15 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693606/ [6693608] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_unmap} time: 00:15:25 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693608/ [6693610] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} time: 00:26:02 info: http://pulpito.ceph.com/teuthology-2022-02-18_03:20:03-krbd-master-testing-default-smithi/6693610/