2022-02-19T08:36:36.794 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-19T08:36:36.796 INFO:teuthology.results:seed: '6418' 2022-02-19T08:59:49.770 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-19T08:59:49.799 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-19T09:28:52.046 INFO:teuthology.results:Tests finished! gathering results... 2022-02-19T09:28:53.634 INFO:scrape:Found 27 jobs 2022-02-19T09:28:54.582 INFO:scrape:Found 1 distinct failure reasons 2022-02-19T09:28:54.583 INFO:scrape:Failure: "2022-02-19T09:21:08.076916+0000 mon.a (mon.0) 268 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2022-02-19T09:28:54.583 INFO:scrape:7 jobs: ['6696003', '6695990', '6696010', '6695998', '6696006', '6696002', '6696008'] 2022-02-19T09:28:54.584 INFO:scrape:suites intersection: ['conf'] 2022-02-19T09:28:54.584 INFO:scrape:suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/object-map', 'install/ceph', 'krbd/fsx/{ceph/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode$/{crc-rxbounce}', 'ms_mode/crc$/{crc}', 'ms_mode/legacy$/{legacy-rxbounce}', 'ms_mode/legacy$/{legacy}', 'ms_mode/secure', 'msgr-failures/many', 'objectstore/bluestore-bitmap', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-1-client}', 'tasks/krbd_huge_osdmap}', 'tasks/krbd_udev_netns}', 'tasks/rbd_fio}', 'tasks/rbd_workunit_suites_fsstress_ext4}', 'tasks/rbd_workunit_suites_fsstress}', 'tasks/rbd_workunit_suites_fsx}'] 2022-02-19T09:28:54.585 INFO:scrape: 2022-02-19T09:28:54.586 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 20 Fail ================================================================= [6695990] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netns} ----------------------------------------------------------------- time: 00:19:29 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695990/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d63e1e5dc2744b8b9dc7081f74fa074e "2022-02-19T09:07:58.404482+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6695998] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_huge_osdmap} ----------------------------------------------------------------- time: 00:19:42 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695998/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de1bf589b9ff4ae5a3dae7b986030178 "2022-02-19T09:15:28.383951+0000 mon.a (mon.0) 222 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6696002] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:19:26 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696002/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8849145f04254ae8ac50f4d7802db8c0 "2022-02-19T09:20:31.770336+0000 mon.a (mon.0) 275 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6696003] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:46 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696003/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696003/ sentry: https://sentry.ceph.com/organizations/ceph/?query=162af036f34d4093a2f1122b3cb37c06 "2022-02-19T09:21:08.076916+0000 mon.a (mon.0) 268 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6696006] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} ----------------------------------------------------------------- time: 00:19:21 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696006/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696006/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15b26458a77649dabd030775c231e4a1 "2022-02-19T09:12:34.799807+0000 mon.a (mon.0) 182 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6696008] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_fio} ----------------------------------------------------------------- time: 00:18:58 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696008/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696008/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af13ecb223484da6bfe4ddb042f0663c "2022-02-19T09:23:59.286339+0000 mon.a (mon.0) 253 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6696010] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:19:22 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696010/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696010/ sentry: https://sentry.ceph.com/organizations/ceph/?query=969049afa52e4c7a9234402955ddfc82 "2022-02-19T09:14:44.731125+0000 mon.a (mon.0) 179 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [6695987] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_kernel} time: 00:10:27 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695987/ [6695988] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_unmap} time: 00:15:27 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695988/ [6695989] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} time: 00:38:23 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695989/ [6695991] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} time: 00:15:47 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695991/ [6695992] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:20:57 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695992/ [6695993] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_kernel} time: 00:10:18 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695993/ [6695994] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:09:55 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695994/ [6695995] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:12:14 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695995/ [6695996] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} time: 00:21:12 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695996/ [6695997] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_fallocate} time: 00:11:05 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695997/ [6695999] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:18:45 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6695999/ [6696000] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio} time: 00:28:31 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696000/ [6696001] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} time: 00:11:43 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696001/ [6696004] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:24 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696004/ [6696005] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} time: 00:11:37 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696005/ [6696007] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} time: 00:10:53 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696007/ [6696009] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_data_pool} time: 00:22:18 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696009/ [6696011] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} time: 00:15:56 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696011/ [6696012] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:10:05 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696012/ [6696013] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:12:12 info: http://pulpito.ceph.com/dis-2022-02-19_08:35:25-krbd-wip-dis-testing-testing-default-smithi/6696013/