2022-03-30T15:06:22.945 INFO:root:teuthology version: 1.1.0-d877b023 2022-03-30T15:06:22.947 INFO:teuthology.results:subset: '111/120000' 2022-03-30T15:06:22.947 INFO:teuthology.results:seed: '9566' 2022-03-30T15:13:19.423 INFO:root:teuthology version: 1.1.0-d877b023 2022-03-30T15:13:19.427 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-30T19:08:36.301 INFO:teuthology.results:Tests finished! gathering results... 2022-03-30T19:08:36.858 INFO:scrape:Found 7 jobs 2022-03-30T19:08:38.605 INFO:scrape:Found 3 distinct failure reasons 2022-03-30T19:08:38.606 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/concurrent.sh 2022-03-30T19:08:38.607 INFO:scrape:1 jobs: ['6769109'] 2022-03-30T19:08:38.607 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/crc$/{crc-rxbounce}', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2022-03-30T19:08:38.608 INFO:scrape: 2022-03-30T19:08:38.608 INFO:scrape:Failure: "2022-03-30T15:32:24.047489+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-03-30T19:08:38.609 INFO:scrape:1 jobs: ['6769106'] 2022-03-30T19:08:38.609 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/legacy$/{legacy-rxbounce}', 'msgr-failures/many', 'tasks/krbd_fallocate}'] 2022-03-30T19:08:38.610 INFO:scrape: 2022-03-30T19:08:38.610 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2022-03-30T19:08:38.610 INFO:scrape:1 jobs: ['6769107'] 2022-03-30T19:08:38.611 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2022-03-30T19:08:38.611 INFO:scrape: 2022-03-30T19:08:38.612 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [6769106] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_fallocate} ----------------------------------------------------------------- time: 00:20:30 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769106/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d95b6f7ffa74d348377f5f5a0655d06 "2022-03-30T15:32:24.047489+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6769107] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769107/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769107/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2db06debcaa4f6eb30ae37d4fe04d93 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6769109] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 03:45:00 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769109/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769109/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b90e9f24d3f1469696e146a195e4ef43 Command failed (workunit test rbd/concurrent.sh) on smithi174 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=406c2368e3b99f606b7749ff951eb50f2a2d612e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh' Pass ================================================================= [6769104] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_kernel} time: 00:11:30 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769104/ [6769105] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} time: 00:28:40 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769105/ [6769108] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} time: 00:41:16 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769108/ [6769110] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} time: 00:22:30 info: http://pulpito.ceph.com/dis-2022-03-30_15:05:30-krbd-quincy-testing-default-smithi/6769110/