2022-02-02T01:19:25.092 INFO:root:teuthology version: 1.1.0-0f634dc1 2022-02-02T01:19:25.094 INFO:teuthology.results:seed: '8' 2022-02-02T01:45:50.841 INFO:root:teuthology version: 1.1.0-0f634dc1 2022-02-02T01:45:50.874 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-02T06:41:12.799 INFO:teuthology.results:Tests finished! gathering results... 2022-02-02T06:41:15.850 INFO:scrape:Found 30 jobs 2022-02-02T06:41:21.066 INFO:scrape:Found 4 distinct failure reasons 2022-02-02T06:41:21.067 INFO:scrape:Failure: "2022-02-02T01:57:24.813907+0000 mon.a (mon.0) 185 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-02-02T06:41:21.068 INFO:scrape:4 jobs: ['6654024', '6654040', '6654020', '6654018'] 2022-02-02T06:41:21.068 INFO:scrape:suites intersection: ['clusters/fixed-3', 'conf', 'msgr-failures/many'] 2022-02-02T06:41:21.069 INFO:scrape:suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/legacy$/{legacy-rxbounce}', 'ms_mode/legacy$/{legacy}', 'ms_mode/secure', 'msgr-failures/many', 'tasks/krbd_udev_symlinks}', 'tasks/rbd_image_read}', 'tasks/rbd_kernel}', 'tasks/rbd_workunit_suites_ffsb}'] 2022-02-02T06:41:21.070 INFO:scrape: 2022-02-02T06:41:21.070 INFO:scrape:Failure: Command failed on smithi146 with status 2: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludely_dfxr_ -r -- '-g auto -g blockdev -x clone'" 2022-02-02T06:41:21.071 INFO:scrape:2 jobs: ['6653990', '6654028'] 2022-02-02T06:41:21.071 INFO:scrape:suites intersection: ['conf', 'krbd/singleton/{bluestore-bitmap', 'tasks/rbd_xfstests}'] 2022-02-02T06:41:21.072 INFO:scrape:suites union: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{crc-rxbounce}', 'ms_mode$/{crc}', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_xfstests}'] 2022-02-02T06:41:21.072 INFO:scrape: 2022-02-02T06:41:21.073 INFO:scrape:Failure: SSH connection to smithi183 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-02-02T06:41:21.073 INFO:scrape:1 jobs: ['6654033'] 2022-02-02T06:41:21.074 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/legacy$/{legacy}', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2022-02-02T06:41:21.074 INFO:scrape: 2022-02-02T06:41:21.074 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2022-02-02T06:41:21.075 INFO:scrape:1 jobs: ['6653992'] 2022-02-02T06:41:21.075 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2022-02-02T06:41:21.076 INFO:scrape: 2022-02-02T06:41:21.077 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/ failed: 7 dead: 1 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [6653990] krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc} msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:24:13 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653990/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b15005d7e08e45f4baf612285ac4c5ca Command failed on smithi146 with status 2: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludely_dfxr_ -r -- '-g auto -g blockdev -x clone'" [6653992] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653992/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71259fa04af04c87b0841cabe8658a15 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6654018] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:19:53 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654018/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654018/ sentry: https://sentry.ceph.com/organizations/ceph/?query=626b616217e241e3b158cfc8fe9abbb3 "2022-02-02T02:01:35.242822+0000 mon.a (mon.0) 250 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6654020] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} ----------------------------------------------------------------- time: 00:19:43 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654020/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654020/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85bc7b103317488eb4edcc3edc08178b "2022-02-02T02:03:24.032087+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6654024] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:44 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654024/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654024/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28976f37cdaf407eab219c48569bf424 "2022-02-02T01:57:24.813907+0000 mon.a (mon.0) 185 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6654028] krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc-rxbounce} msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:50:05 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654028/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654028/ sentry: https://sentry.ceph.com/organizations/ceph/?query=811735aeefe74a578026250acf8f27c2 Command failed on smithi149 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludeqdv9av3s -r -- '-g auto -g blockdev -x clone'" [6654040] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:21:13 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654040/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654040/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46e3104e4f474de9aa01ba435bf23948 "2022-02-02T02:15:47.766335+0000 mon.a (mon.0) 309 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log Dead ================================================================= [6654033] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 03:03:44 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654033/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654033/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c80945c72f9c4710b98e8cfbb2a06d58 SSH connection to smithi183 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' Pass ================================================================= [6653994] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:34 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653994/ [6653996] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_discard} time: 00:10:16 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653996/ [6653998] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} time: 00:12:07 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6653998/ [6654000] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} time: 00:12:05 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654000/ [6654002] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} time: 00:23:41 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654002/ [6654004] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:16:58 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654004/ [6654006] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netns} time: 00:12:09 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654006/ [6654008] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:20:15 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654008/ [6654010] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:20:28 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654010/ [6654012] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_fallocate} time: 00:11:36 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654012/ [6654014] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:13:54 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654014/ [6654016] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_discard} time: 00:09:52 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654016/ [6654022] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:13 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654022/ [6654026] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} time: 00:25:24 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654026/ [6654030] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:15:00 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654030/ [6654032] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} time: 00:12:26 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654032/ [6654034] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_unmap} time: 00:16:42 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654034/ [6654035] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} time: 00:29:55 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654035/ [6654036] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_exclusive_option} time: 00:12:56 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654036/ [6654037] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} time: 00:11:01 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654037/ [6654038] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_namespaces} time: 00:13:36 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654038/ [6654039] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:23:12 info: http://pulpito.ceph.com/dis-2022-02-02_01:05:06-krbd-wip-krbd-rxbounce-option-testing-default-smithi/6654039/