2022-05-16T08:38:12.976 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-16T08:38:12.977 INFO:teuthology.results:seed: '903' 2022-05-16T09:07:05.596 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-16T09:07:05.613 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-16T13:46:27.985 INFO:teuthology.results:Tests finished! gathering results... 2022-05-16T13:46:29.343 INFO:scrape:Found 20 jobs 2022-05-16T13:46:32.130 INFO:scrape:Found 2 distinct failure reasons 2022-05-16T13:46:32.131 INFO:scrape:Failure: "2022-05-16T09:27:21.154683+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2022-05-16T13:46:32.131 INFO:scrape:11 jobs: ['6835790', '6835786', '6835793', '6835791', '6835780', '6835798', '6835795', '6835782', '6835797', '6835788', '6835781'] 2022-05-16T13:46:32.132 INFO:scrape:suites intersection: ['clusters/fixed-3', 'conf', 'msgr-failures/many'] 2022-05-16T13:46:32.132 INFO:scrape:suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/crc$/{crc-rxbounce}', 'ms_mode/crc$/{crc}', 'ms_mode/legacy$/{legacy-rxbounce}', 'ms_mode/legacy$/{legacy}', 'ms_mode/secure', 'msgr-failures/many', 'tasks/krbd_exclusive_option}', 'tasks/krbd_fallocate}', 'tasks/krbd_udev_netlink_enobufs}', 'tasks/rbd_concurrent}', 'tasks/rbd_huge_tickets}', 'tasks/rbd_image_read}', 'tasks/rbd_workunit_suites_dbench}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_suites_iozone}'] 2022-05-16T13:46:32.133 INFO:scrape: 2022-05-16T13:46:32.133 INFO:scrape:Failure: Command failed on smithi045 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude0buk2613 -r -- '-g auto -g blockdev -x clone'" 2022-05-16T13:46:32.134 INFO:scrape:2 jobs: ['6835787', '6835779'] 2022-05-16T13:46:32.134 INFO:scrape:suites intersection: ['conf', 'krbd/singleton/{bluestore-bitmap', 'tasks/rbd_xfstests}'] 2022-05-16T13:46:32.134 INFO:scrape:suites union: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{crc}', 'ms_mode$/{legacy}', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_xfstests}'] 2022-05-16T13:46:32.135 INFO:scrape: 2022-05-16T13:46:32.136 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/ failed: 13 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [6835779] krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc} msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:26:26 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835779/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=024d5f296f2a4aea8c8fb393755b068b Command failed on smithi100 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude2ytjztyb -r -- '-g auto -g blockdev -x clone'" [6835780] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} ----------------------------------------------------------------- time: 00:21:59 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835780/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=407a91c7833040b5abbaf7f312da522b "2022-05-16T09:13:10.059435+0000 mon.a (mon.0) 223 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835781] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:21:49 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835781/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4cea5d072fea403b913472874be629b4 "2022-05-16T09:12:45.647879+0000 mon.a (mon.0) 283 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835782] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_fallocate} ----------------------------------------------------------------- time: 00:22:07 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835782/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=351ad74280144ee69dee0b4f37f2d8a5 "2022-05-16T09:14:19.376763+0000 mon.a (mon.0) 253 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835786] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:21:54 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835786/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835786/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27392cba42bb4fa58131f4c4fbc4e0ef "2022-05-16T09:17:37.168759+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835787] krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:47:58 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835787/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ec0e7a62f03490e9f2c971d8ffbfa2b Command failed on smithi045 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude0buk2613 -r -- '-g auto -g blockdev -x clone'" [6835788] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:22:02 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835788/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=465eaef34ced4db18a8b42c18f0a2962 "2022-05-16T09:20:04.485851+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835790] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:21:52 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835790/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8cb13c011011471d96346d7efce8baa5 "2022-05-16T09:27:21.154683+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835791] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:21:54 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835791/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ee1a330d02b487196beedfffaa679b8 "2022-05-16T09:29:42.230333+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835793] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:22:04 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835793/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0038d5fa81324680ac57a5c21bb43084 "2022-05-16T09:20:41.482008+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6835795] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:22:17 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835795/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835795/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55d016c86c044db6831786a68a134248 "2022-05-16T09:33:46.839986+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835797] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:21:56 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835797/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7edf16a1d9644bdb39fcc51379d9a80 "2022-05-16T09:37:14.707469+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6835798] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:22:02 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835798/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e213335eb934b638317e56004d0bb3a "2022-05-16T09:39:07.806551+0000 mon.a (mon.0) 286 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log Pass ================================================================= [6835783] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netns} time: 00:14:34 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835783/ [6835784] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:13:03 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835784/ [6835785] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} time: 00:13:22 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835785/ [6835789] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:16:58 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835789/ [6835792] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:21:25 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835792/ [6835794] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} time: 00:14:49 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835794/ [6835796] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:17:11 info: http://pulpito.ceph.com/dis-2022-05-16_08:37:25-krbd-quincy-wip-fix-linger-resend-default-smithi/6835796/