2021-03-08T18:41:43.727 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-08T18:41:43.729 INFO:teuthology.results:subset: '245/300' 2021-03-08T18:41:43.730 INFO:teuthology.results:seed: '8667' 2021-03-08T23:43:14.760 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-08T23:43:14.791 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-09T11:44:14.503 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-03-09T11:44:37.280 INFO:scrape:Found 196 jobs 2021-03-09T11:44:39.059 INFO:scrape:Found 5 distinct failure reasons 2021-03-09T11:44:39.060 INFO:scrape:Dead: b'2021-03-09T11:41:44.548 DEBUG:teuthology.exit:Finished running handlers' 2021-03-09T11:44:39.060 INFO:scrape:3 jobs: ['5946768', '5946744', '5946674'] 2021-03-09T11:44:39.061 INFO:scrape:suites intersection: ['2-cluster/{fix-2', '3-supported-random-distro$/{rhel_8}', '4-pool/cache', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] 2021-03-09T11:44:39.061 INFO:scrape:suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{rhel_8}', '4-pool/cache', '5-cache-mode/rwl', '5-cache-mode/ssd', '6-workloads/fio}', '6-workloads/qemu_xfstests}', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] 2021-03-09T11:44:39.062 INFO:scrape: 2021-03-09T11:44:39.062 INFO:scrape:Failure: "2021-03-08T22:57:58.944430+0000 osd.1 (osd.1) 3 : cluster [WRN] slow request osd_op(client.4451.0:150386 4.2 4:4b09a614:::rbd_data.1189f4bf808c.000000000000047a:head [stat out=16b,write 2998272~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected e37) initiated 2021-03-08T22:57:28.609222+0000 currently waiting for sub ops" in cluster log 2021-03-09T11:44:39.063 INFO:scrape:1 jobs: ['5946702'] 2021-03-09T11:44:39.063 INFO:scrape:suites: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/ec-cache-pool', 'rbd/qemu/{cache/writearound', 'supported-random-distro$/{rhel_8}', 'workloads/qemu_bonnie}'] 2021-03-09T11:44:39.064 INFO:scrape: 2021-03-09T11:44:39.064 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=76d76b451048ab3dfd44edc3eb255c297d22a641 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2021-03-09T11:44:39.065 INFO:scrape:1 jobs: ['5946630'] 2021-03-09T11:44:39.065 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] 2021-03-09T11:44:39.065 INFO:scrape: 2021-03-09T11:44:39.066 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=76d76b451048ab3dfd44edc3eb255c297d22a641 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 2021-03-09T11:44:39.066 INFO:scrape:1 jobs: ['5946623'] 2021-03-09T11:44:39.067 INFO:scrape:suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{rhel_8}', '4-pool/cache', '5-cache-mode/ssd', '6-workloads/c_api_tests_with_defaults}', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] 2021-03-09T11:44:39.067 INFO:scrape: 2021-03-09T11:44:39.068 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi192 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=76d76b451048ab3dfd44edc3eb255c297d22a641 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2021-03-09T11:44:39.068 INFO:scrape:1 jobs: ['5946606'] 2021-03-09T11:44:39.069 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-workunit-config-key}'] 2021-03-09T11:44:39.069 INFO:scrape: