2021-09-29T08:41:04.880 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-29T08:41:04.882 INFO:teuthology.results:seed: '5539' 2021-09-29T09:27:33.329 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-29T09:27:33.358 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-29T13:25:53.616 INFO:teuthology.results:Tests finished! gathering results... 2021-09-29T13:25:57.833 INFO:scrape:Found 53 jobs 2021-09-29T13:26:02.184 INFO:scrape:Found 3 distinct failure reasons 2021-09-29T13:26:02.185 INFO:scrape:Failure: "2021-09-29T09:20:48.220038+0000 mon.a (mon.0) 94 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2021-09-29T13:26:02.185 INFO:scrape:22 jobs: ['6413309', '6413325', '6413327', '6413329', '6413289', '6413296', '6413300', '6413303', '6413285', '6413316', '6413305', '6413320', '6413333', '6413301', '6413326', '6413295', '6413288', '6413313', '6413332', '6413314', '6413302', '6413308'] 2021-09-29T13:26:02.186 INFO:scrape:suites intersection: ['conf'] 2021-09-29T13:26:02.186 INFO:scrape:suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/no-object-map', 'features/object-map', 'install/ceph', 'krbd/fsx/{ceph/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{legacy}', 'ms_mode$/{prefer-crc}', 'ms_mode$/{secure}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/few', 'msgr-failures/many', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-1-client}', 'tasks/fsx-3-client}', 'tasks/krbd_data_pool}', 'tasks/krbd_huge_osdmap}', 'tasks/krbd_udev_netlink_enobufs}', 'tasks/rbd_concurrent}', 'tasks/rbd_huge_tickets}', 'tasks/rbd_image_read}', 'tasks/rbd_map_snapshot_io}', 'tasks/rbd_map_unmap}', 'tasks/rbd_simple_big}', 'tasks/rbd_workunit_kernel_untar_build}', 'tasks/rbd_workunit_suites_dbench}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_suites_iozone}', 'tasks/rbd_workunit_trivial_sync}', 'tasks/rbd_xfstests}'] 2021-09-29T13:26:02.187 INFO:scrape: 2021-09-29T13:26:02.187 INFO:scrape:Failure: Command failed on smithi042 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludej3pi9s3i -r -- '-g auto -g blockdev -x clone'" 2021-09-29T13:26:02.188 INFO:scrape:1 jobs: ['6413282'] 2021-09-29T13:26:02.188 INFO:scrape:suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{prefer-crc}', 'msgr-failures/few', 'tasks/rbd_xfstests}'] 2021-09-29T13:26:02.188 INFO:scrape: 2021-09-29T13:26:02.189 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2021-09-29T13:26:02.189 INFO:scrape:1 jobs: ['6413283'] 2021-09-29T13:26:02.190 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2021-09-29T13:26:02.190 INFO:scrape: 2021-09-29T13:26:02.192 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi ================================================================= info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/ failed: 24 dead: 0 running: 0 waiting: 0 queued: 0 passed: 29 Fail ================================================================= [6413282] krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:31:57 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413282/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413282/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86835344499a4a9996f84c0e2e4729d1 Command failed on smithi042 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludej3pi9s3i -r -- '-g auto -g blockdev -x clone'" [6413283] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413283/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413283/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8fe191e38c994b6c8d5bb62e3b665d2c Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6413285] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_kernel_untar_build} ----------------------------------------------------------------- time: 00:19:35 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413285/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ffda0d71575449e8280f2b1cbfc03fb "2021-09-29T09:14:14.179696+0000 mon.a (mon.0) 263 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413288] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:19:26 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413288/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90a44bdb5c4249a58f02760fb474456c "2021-09-29T09:11:08.214368+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413289] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:19:26 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413289/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413289/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b1f3a17b2834c7683e271b3314999e1 "2021-09-29T09:24:27.538324+0000 mon.a (mon.0) 292 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413295] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:21:39 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413295/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68ea3ef7010d4b628459f87859a07ac7 "2021-09-29T09:30:54.857435+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413296] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:18:52 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413296/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07ea9e2206d64bf6a6bdc197c91569e0 "2021-09-29T09:23:34.309178+0000 mon.a (mon.0) 242 : cluster [WRN] Health check failed: 5 osds down (OSD_DOWN)" in cluster log [6413300] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413300/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413300/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57f731175a874743a93f316ea6abe18b "2021-09-29T09:12:13.041743+0000 mon.a (mon.0) 125 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413301] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:21:48 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413301/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413301/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e50651cc080b49f29291d194c73b9495 "2021-09-29T09:28:28.783336+0000 mon.a (mon.0) 259 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413302] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:21:17 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413302/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58a62c91db9747959874006f31eb2a15 "2021-09-29T09:31:17.963242+0000 mon.a (mon.0) 332 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413303] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_simple_big} ----------------------------------------------------------------- time: 00:23:11 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413303/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413303/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb87bf5470b24f00b98fca9cd6b218a7 "2021-09-29T09:20:08.091937+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413305] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:19:45 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413305/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413305/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ede912a210444fe981282c1a79ad40c "2021-09-29T09:22:32.557119+0000 mon.a (mon.0) 290 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413308] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_unmap} ----------------------------------------------------------------- time: 00:21:14 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413308/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413308/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d13104f9699a44de85f0ce5d1e8e18dd "2021-09-29T09:16:18.868693+0000 mon.a (mon.0) 141 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413309] krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 00:19:28 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413309/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413309/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d970d6de26584318b44d7a1d865c6996 "2021-09-29T09:20:48.220038+0000 mon.a (mon.0) 94 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413313] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:22:29 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413313/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a888ac998f36486c8fd96641d6ac96c0 "2021-09-29T09:28:25.075157+0000 mon.a (mon.0) 247 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6413314] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:22:15 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413314/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413314/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e1539b02b0d4e619c58b65dec2314e2 "2021-09-29T09:28:14.736055+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413316] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:22:42 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413316/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d30ff8d47b5455ea2f8f3f05c034ad0 "2021-09-29T09:29:58.451899+0000 mon.a (mon.0) 222 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413320] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} ----------------------------------------------------------------- time: 00:22:52 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413320/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413320/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7f4334e111e45df97d41f42f2dcf610 "2021-09-29T09:20:01.469968+0000 mon.a (mon.0) 115 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413325] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:21:58 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413325/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413325/ sentry: https://sentry.ceph.com/organizations/ceph/?query=337d46af3c8e4c58abcc8230e50e52b9 "2021-09-29T09:30:34.439053+0000 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6413326] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:19:31 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413326/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5099a3f2d134e668b323e80f21388ad "2021-09-29T09:30:21.288727+0000 mon.a (mon.0) 149 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413327] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:18:37 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413327/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77e1bd5210cb4aa199836269f4e0ce68 "2021-09-29T09:40:31.395434+0000 mon.a (mon.0) 261 : cluster [WRN] Health check failed: 5 osds down (OSD_DOWN)" in cluster log [6413329] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_data_pool} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413329/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413329/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4cc9653b248e48d7b77d23dd59de55a4 "2021-09-29T09:32:37.801590+0000 mon.a (mon.0) 189 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413332] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:12 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413332/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43b66fabd4094b248796971ff4c70455 "2021-09-29T09:37:49.264113+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6413333] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:19:18 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413333/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ccffbd1473c84239adc297279c10769a "2021-09-29T09:40:33.010093+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [6413284] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_exclusive_option} time: 00:12:59 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413284/ [6413286] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} time: 00:11:55 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413286/ [6413287] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:19 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413287/ [6413290] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_kernel} time: 00:13:39 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413290/ [6413291] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:18:12 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413291/ [6413292] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:50 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413292/ [6413293] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} time: 00:23:48 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413293/ [6413294] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_fallocate} time: 00:14:59 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413294/ [6413297] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} time: 00:37:46 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413297/ [6413298] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:15:49 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413298/ [6413299] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_concurrent} time: 00:17:14 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413299/ [6413304] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} time: 00:19:14 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413304/ [6413306] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:22:03 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413306/ [6413307] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} time: 00:14:17 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413307/ [6413310] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} time: 00:13:30 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413310/ [6413311] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:17:38 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413311/ [6413312] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:15:01 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413312/ [6413315] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:21:41 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413315/ [6413317] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:13:17 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413317/ [6413318] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:29:05 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413318/ [6413319] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} time: 00:19:56 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413319/ [6413321] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:39:40 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413321/ [6413322] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:14:25 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413322/ [6413323] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} time: 00:10:46 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413323/ [6413324] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} time: 00:35:13 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413324/ [6413328] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_fio} time: 00:28:40 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413328/ [6413330] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:16:34 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413330/ [6413331] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} time: 00:15:49 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413331/ [6413334] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:10:15 info: http://pulpito.ceph.com/dis-2021-09-29_08:40:49-krbd-master-wip-rxbounce-basic-smithi/6413334/