2023-07-11T16:07:32.089 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-11T16:07:32.090 INFO:teuthology.results:seed: '2070'
2023-07-11T16:08:22.758 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-11T16:08:22.762 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-11T16:38:24.659 INFO:teuthology.results:Tests finished! gathering results...
2023-07-11T16:38:24.942 INFO:scrape:Found 8 jobs
2023-07-11T16:38:25.085 INFO:scrape:Found 1 distinct failure reasons
2023-07-11T16:38:25.086 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi184 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2023-07-11T16:38:25.086 INFO:scrape:8 jobs: ['7333915', '7333918', '7333920', '7333916', '7333921', '7333922', '7333917', '7333919']
2023-07-11T16:38:25.087 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}']
2023-07-11T16:38:25.087 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}']
2023-07-11T16:38:25.087 INFO:scrape:
2023-07-11T16:38:25.088 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/
failed:  8
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7333915]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:18:31
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333915/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333915/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d7c1436aa510494893ab7d37679e7773

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi184 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333916]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:13:16
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333916/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b6fbc0f588a414e9331549c9c57affa

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi117 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333917]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:21:21
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333917/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b76cf8a492448c3a51e2e0ba01b653d

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi033 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333918]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:11:19
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333918/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333918/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe430d785f9a41d9924439a72666bf15

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi145 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333919]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:22:11
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333919/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=391a6a37282b4ff5aae0c3a8f25ed432

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi177 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333920]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:20:50
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333920/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f0ea6abe6bf43ed9a2c0fdacbb0093a

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi167 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333921]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:13:34
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333921/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333921/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd571f98b4c04fab8c17cc04ac48f994

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi203 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7333922]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:13:32
info:   http://pulpito.ceph.com/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333922/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-11_16:07:26-rbd:mirror-wip-choffman-61707-distro-default-smithi/7333922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1259db918e414f30808a05bea8f657aa

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi160 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0798ecd7104f8ca00e8c2eda5efb2ba1f123998b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'