2023-07-12T14:46:17.174 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-12T14:46:17.175 INFO:teuthology.results:seed: '9412'
2023-07-12T14:49:24.119 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-12T14:49:24.124 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-12T15:22:26.810 INFO:teuthology.results:Tests finished! gathering results...
2023-07-12T15:22:27.096 INFO:scrape:Found 8 jobs
2023-07-12T15:22:27.317 INFO:scrape:Found 1 distinct failure reasons
2023-07-12T15:22:27.318 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi149 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2023-07-12T15:22:27.318 INFO:scrape:8 jobs: ['7335366', '7335364', '7335368', '7335365', '7335370', '7335363', '7335367', '7335369']
2023-07-12T15:22:27.319 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}']
2023-07-12T15:22:27.319 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}']
2023-07-12T15:22:27.319 INFO:scrape:
2023-07-12T15:22:27.321 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/
failed:  8
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7335363]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:24:25
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335363/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335363/
sentry: https://sentry.ceph.com/organizations/ceph/?query=635d08a91f4548808a2c4116fffbad76

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi153 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335364]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:18:28
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335364/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335364/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40552aec3a3a4cd7ae002ff80998a46a

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi163 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335365]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:20:05
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335365/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335365/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6b4a259d580041ea8b73847b18b829f2

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi196 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335366]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:11:29
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335366/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335366/
sentry: https://sentry.ceph.com/organizations/ceph/?query=62e19cd4412548dda1abfc8620d5b0d7

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi149 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335367]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:18:59
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335367/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335367/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ff0d815fe374d52ab62f2d8a60628ca

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi092 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335368]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:18:44
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335368/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335368/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e880937b209d42059671c7cd0bbc5abb

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi079 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335369]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:13:16
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335369/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335369/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d6f8d48d1186421e9cd8740bc15e6edd

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi088 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'

[7335370]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock}
-----------------------------------------------------------------
time:   00:21:16
info:   http://pulpito.ceph.com/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335370/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-07-12_14:43:48-rbd:mirror-wip-choffman-61707-distro-default-smithi/7335370/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e5af4a0a222415e9c52d99b4b3294ae

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi123 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=850dc801e7256ef7f299d950ec38bf47e043533c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho
    t.sh'