2022-03-07T10:36:01.949 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-07T10:36:01.952 INFO:teuthology.results:seed: '266' 2022-03-07T11:06:06.980 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-07T11:06:07.000 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-07T11:27:08.615 INFO:teuthology.results:Tests finished! gathering results... 2022-03-07T11:27:09.728 INFO:scrape:Found 18 jobs 2022-03-07T11:27:11.759 INFO:scrape:Found 2 distinct failure reasons 2022-03-07T11:27:11.761 INFO:scrape:Failure: Command failed on smithi070 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916' 2022-03-07T11:27:11.761 INFO:scrape:6 jobs: ['6724842', '6724829', '6724832', '6724845', '6724837', '6724840'] 2022-03-07T11:27:11.762 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-07T11:27:11.762 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-07T11:27:11.764 INFO:scrape: 2022-03-07T11:27:11.765 INFO:scrape:Failure: 'cluster3' 2022-03-07T11:27:11.765 INFO:scrape:12 jobs: ['6724838', '6724839', '6724835', '6724830', '6724828', '6724831', '6724836', '6724843', '6724841', '6724834', '6724844', '6724833'] 2022-03-07T11:27:11.766 INFO:scrape:suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-07T11:27:11.766 INFO:scrape:suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-07T11:27:11.767 INFO:scrape: 2022-03-07T11:27:11.768 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 18 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6724828] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:08:14 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724828/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a59dd2eed5a946e1b1ae77f8f521372d 'cluster3' [6724829] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:11:37 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724829/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0dc07ba42136443bb785fa03e2382bc5 Command failed on smithi070 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916' [6724830] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:07:51 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724830/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdfac2935003440498294808b1f1f869 'cluster3' [6724831] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:08:01 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724831/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724831/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27aee3c2949740008cd2a86002a0623a 'cluster3' [6724832] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:09:09 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724832/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db62d5efb29c4a7c85a446a007d03ce3 Command failed on smithi148 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916' [6724833] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:08:21 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724833/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724833/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7417c4959bb40128cab651069c38507 'cluster3' [6724834] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:10:22 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724834/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddd69e7e6c92459b86c259d58dba9794 'cluster3' [6724835] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:07:55 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724835/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724835/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdfc2e058e384356a8cfa6f85f87708b 'cluster3' [6724836] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:07:20 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724836/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0920fd9dc634801891282492f7d5b10 'cluster3' [6724837] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724837/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a66033469c084d50a486c6a70b38feff Command failed on smithi027 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916' [6724838] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:10:33 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724838/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e765e6611404c61b4f0a56c90783b0d 'cluster3' [6724839] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:10:26 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724839/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724839/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b105cc95b7c4266ba6f94aa304cc836 'cluster3' [6724840] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:11:31 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724840/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b90d19fbf8647aea0406b45486cea21 Command failed on smithi016 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916' [6724841] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:10:19 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724841/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6eb27434bbf545c898edecdffb530322 'cluster3' [6724842] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:11:41 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724842/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d328a8b191d44eb89c396ffa0cd429d7 Command failed on smithi070 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916' [6724843] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:09:51 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724843/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d34dea54c8c461db817306bfc4c31a6 'cluster3' [6724844] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:10:41 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724844/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724844/ sentry: https://sentry.ceph.com/organizations/ceph/?query=398ef84ebdc44529bbe09994bbef6f2e 'cluster3' [6724845] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724845/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_10:35:45-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6724845/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce2cf52b7f1543baa6672afc30400929 Command failed on smithi061 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e37086e19ef2ba1c8b6a5843c47d9139b084a916'