2022-05-04T10:03:39.180 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-05-04T10:03:39.181 INFO:teuthology.results:subset: '102/999' 2022-05-04T10:03:39.181 INFO:teuthology.results:seed: '4660' 2022-05-04T10:47:13.220 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-05-04T10:47:13.237 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-04T11:50:17.729 INFO:teuthology.results:Tests finished! gathering results... 2022-05-04T11:50:19.646 INFO:scrape:Found 26 jobs 2022-05-04T11:50:19.813 INFO:scrape:Found 1 distinct failure reasons 2022-05-04T11:50:19.814 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adbeacc93fa86dfcb7ca824a79c980c94b11b6ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2022-05-04T11:50:19.815 INFO:scrape:1 jobs: ['6820427'] 2022-05-04T11:50:19.815 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] 2022-05-04T11:50:19.816 INFO:scrape: 2022-05-04T11:50:19.817 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [6820427] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} ----------------------------------------------------------------- time: 00:11:27 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820427/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820427/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe2ea55f79a14335baa93d8e5508ffaa Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adbeacc93fa86dfcb7ca824a79c980c94b11b6ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _stress.sh' Pass ================================================================= [6820404] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit} time: 00:40:01 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820404/ [6820405] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:19:09 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820405/ [6820406] rbd/singleton/{all/rbd_mirror objectstore/filestore-xfs openstack supported-random-distro$/{rhel_8}} time: 00:46:01 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820406/ [6820407] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} time: 00:50:52 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820407/ [6820408] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} time: 00:56:28 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820408/ [6820409] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-config-key} time: 00:21:13 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820409/ [6820410] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} time: 00:36:50 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820410/ [6820411] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} time: 00:20:08 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820411/ [6820412] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:46:37 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820412/ [6820413] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-policy-none} time: 00:26:28 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820413/ [6820414] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:55:16 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820414/ [6820415] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:22:17 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820415/ [6820416] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-ha-workunit} time: 00:14:36 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820416/ [6820417] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:46:27 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820417/ [6820418] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-bootstrap-workunit} time: 00:08:01 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820418/ [6820419] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} time: 00:40:41 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820419/ [6820420] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-bootstrap-workunit} time: 00:09:09 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820420/ [6820421] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:03:26 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820421/ [6820422] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:19:44 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820422/ [6820423] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/rbd_mirror} time: 00:52:38 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820423/ [6820424] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} time: 00:35:46 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820424/ [6820425] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:25:48 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820425/ [6820426] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:25:17 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820426/ [6820428] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:21:09 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820428/ [6820429] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:52:56 info: http://pulpito.ceph.com/dis-2022-05-04_10:02:59-rbd-wip-dis-testing-distro-default-smithi/6820429/