2022-02-04T10:44:08.311 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-04T10:44:08.312 INFO:teuthology.results:seed: '123' 2022-02-04T11:23:25.429 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-04T11:23:25.434 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-04T12:38:30.865 INFO:teuthology.results:Tests finished! gathering results... 2022-02-04T12:38:33.803 INFO:scrape:Found 33 jobs 2022-02-04T12:38:35.374 INFO:scrape:Found 2 distinct failure reasons 2022-02-04T12:38:35.375 INFO:scrape:Failure: "2022-02-04T11:41:24.445112+0000 mon.a (mon.0) 276 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2022-02-04T12:38:35.376 INFO:scrape:10 jobs: ['6661327', '6661324', '6661334', '6661313', '6661321', '6661311', '6661336', '6661332', '6661307', '6661315'] 2022-02-04T12:38:35.376 INFO:scrape:suites intersection: ['clusters/fixed-3', 'conf', 'msgr-failures/many'] 2022-02-04T12:38:35.377 INFO:scrape:suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/crc$/{crc-rxbounce}', 'ms_mode/crc$/{crc}', 'ms_mode/legacy$/{legacy-rxbounce}', 'ms_mode/secure', 'msgr-failures/many', 'tasks/krbd_huge_osdmap}', 'tasks/krbd_udev_netns}', 'tasks/krbd_udev_symlinks}', 'tasks/rbd_concurrent}', 'tasks/rbd_map_snapshot_io}', 'tasks/rbd_workunit_suites_ffsb}', 'tasks/rbd_workunit_suites_fsstress}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_suites_iozone}'] 2022-02-04T12:38:35.377 INFO:scrape: 2022-02-04T12:38:35.378 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2022-02-04T12:38:35.378 INFO:scrape:1 jobs: ['6661304'] 2022-02-04T12:38:35.379 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2022-02-04T12:38:35.379 INFO:scrape: 2022-02-04T12:38:35.381 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/ failed: 11 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [6661304] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661304/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661304/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ed2c3b35ea049b0bedc42dae280ff32 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6661307] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_huge_osdmap} ----------------------------------------------------------------- time: 00:19:09 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661307/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661307/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad40d4e26c064d05b99c8521d30919e1 "2022-02-04T11:15:09.341368+0000 mon.a (mon.0) 281 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6661311] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:19:39 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661311/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661311/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2788afea94204ea29fe11bc2e278f795 "2022-02-04T11:20:33.048915+0000 mon.a (mon.0) 301 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6661313] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:19:10 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661313/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=218d2b8bb4414c169fc73964993fb29a "2022-02-04T11:19:49.308948+0000 mon.a (mon.0) 287 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6661315] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:19:35 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661315/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661315/ sentry: https://sentry.ceph.com/organizations/ceph/?query=230435770f184ca2a010d7d38fffe71e "2022-02-04T11:20:28.122037+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6661321] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:18:51 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661321/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce0f808dc2794450a82e49cc82069d9d "2022-02-04T11:21:09.384287+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6661324] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} ----------------------------------------------------------------- time: 01:21:50 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661324/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661324/ "2022-02-04T11:51:38.935784+0000 osd.5 (osd.5) 3 : cluster [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'rbd' : 1 ])" in cluster log [6661327] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661327/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9bffffdbfb749dcb5b1353596309ae4 "2022-02-04T11:41:24.445112+0000 mon.a (mon.0) 276 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6661332] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netns} ----------------------------------------------------------------- time: 00:20:12 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661332/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a609fce33544480aa04f4475e6d2d102 "2022-02-04T11:54:09.992136+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6661334] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:59 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661334/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661334/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bb6cb9299b04b03a5bd369e4baf13e1 "2022-02-04T11:54:19.504588+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6661336] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_snapshot_io} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661336/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ebbd95fa70b462b9d79090d86d190e9 "2022-02-04T11:53:58.913428+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log Pass ================================================================= [6661305] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} time: 00:11:52 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661305/ [6661306] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:22 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661306/ [6661308] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:23:19 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661308/ [6661309] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netlink_enobufs} time: 00:15:06 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661309/ [6661310] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:23:24 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661310/ [6661312] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:15 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661312/ [6661314] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netns} time: 00:11:28 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661314/ [6661316] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:20:37 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661316/ [6661317] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:19:27 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661317/ [6661318] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:09:55 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661318/ [6661319] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_data_pool} time: 00:17:56 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661319/ [6661320] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} time: 00:11:29 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661320/ [6661322] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio} time: 00:27:55 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661322/ [6661323] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:15:58 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661323/ [6661325] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} time: 00:36:43 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661325/ [6661326] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netns} time: 00:11:40 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661326/ [6661328] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_huge_tickets} time: 00:10:25 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661328/ [6661329] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} time: 00:15:32 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661329/ [6661330] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_data_pool} time: 00:22:41 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661330/ [6661331] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} time: 00:11:42 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661331/ [6661333] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} time: 00:17:57 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661333/ [6661335] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:17:15 info: http://pulpito.ceph.com/dis-2022-02-04_10:43:32-krbd-wip-krbd-rxbounce-option-wip-rxbounce-default-smithi/6661335/