2024-01-08T20:10:00.922 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-08T20:10:00.922 INFO:teuthology.results:seed: '8793'
2024-01-08T20:10:45.722 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-08T20:10:45.726 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-01-08T20:45:47.922 INFO:teuthology.results:Tests finished! gathering results...
2024-01-08T20:45:48.102 INFO:scrape:Found 5 jobs
2024-01-08T20:45:48.650 INFO:scrape:Found 1 distinct failure reasons
2024-01-08T20:45:48.651 INFO:scrape:Failure: Command failed (workunit test rbd/compare_mirror_images.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e5fc98c94b1032c3d86e9def4c33387b8fa53991 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh'
2024-01-08T20:45:48.651 INFO:scrape:5 jobs: ['7509146', '7509148', '7509144', '7509147', '7509145']
2024-01-08T20:45:48.651 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}']
2024-01-08T20:45:48.651 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}']
2024-01-08T20:45:48.651 INFO:scrape:
2024-01-08T20:45:48.652 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7509144]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}
-----------------------------------------------------------------
time:   00:25:01
info:   https://pulpito.ceph.com/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509144/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509144/
sentry: https://sentry.ceph.com/organizations/ceph/?query=08e7c096256a475792e0a0c573c26410

    Command failed (workunit test rbd/compare_mirror_images.sh) on smithi159
    with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e5fc98c94b1032c3d86e9def4c33387b8fa53991
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ges.sh'

[7509145]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}
-----------------------------------------------------------------
time:   00:24:57
info:   https://pulpito.ceph.com/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509145/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509145/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b3b0b5ee42354c4dade0835515779d74

    Command failed (workunit test rbd/compare_mirror_images.sh) on smithi093
    with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e5fc98c94b1032c3d86e9def4c33387b8fa53991
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ges.sh'

[7509146]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}
-----------------------------------------------------------------
time:   00:25:07
info:   https://pulpito.ceph.com/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509146/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509146/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a8508dce667c4ef59878c6c73db1c270

    Command failed (workunit test rbd/compare_mirror_images.sh) on smithi176
    with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e5fc98c94b1032c3d86e9def4c33387b8fa53991
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ges.sh'

[7509147]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}
-----------------------------------------------------------------
time:   00:24:49
info:   https://pulpito.ceph.com/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509147/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509147/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c2407ec50964c58a4b4e6ddf321ccef

    Command failed (workunit test rbd/compare_mirror_images.sh) on smithi081
    with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e5fc98c94b1032c3d86e9def4c33387b8fa53991
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ges.sh'

[7509148]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-images-nbd}
-----------------------------------------------------------------
time:   00:23:40
info:   https://pulpito.ceph.com/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509148/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-01-08_20:08:40-rbd:mirror-main-distro-default-smithi/7509148/
sentry: https://sentry.ceph.com/organizations/ceph/?query=32d73d8505b24478a59fa2c271203cfb

    Command failed (workunit test rbd/compare_mirror_images.sh) on smithi137
    with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e5fc98c94b1032c3d86e9def4c33387b8fa53991
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ges.sh'