2023-11-28T19:24:20.214 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5
2023-11-28T19:24:20.214 INFO:teuthology.results:seed: '9671'
2023-11-28T19:26:43.090 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5
2023-11-28T19:26:43.094 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-28T19:55:45.408 INFO:teuthology.results:Tests finished! gathering results...
2023-11-28T19:55:45.529 INFO:scrape:Found 4 jobs
2023-11-28T19:55:45.952 INFO:scrape:Found 2 distinct failure reasons
2023-11-28T19:55:45.952 INFO:scrape:Failure: Command failed (workunit test rbd/mirror_compare.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c60921579da9a8c8d5d71cddc1a39b26073493ca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=krbd RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/mirror_compare.sh'
2023-11-28T19:55:45.953 INFO:scrape:2 jobs: ['7470274', '7470276']
2023-11-28T19:55:45.953 INFO:scrape:suites intersection: ['conf', 'install/ceph', 'rbd:mirror-compare/{clusters/2-node']
2023-11-28T19:55:45.953 INFO:scrape:suites union: ['conf', 'install/ceph', 'rbd:mirror-compare/{clusters/2-node', 'tasks/krbd_mirror_compare}', 'tasks/rbd_nbd_mirror_compare}']
2023-11-28T19:55:45.953 INFO:scrape:
2023-11-28T19:55:45.953 INFO:scrape:Failure: Command failed (workunit test rbd/mirror_compare2.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c60921579da9a8c8d5d71cddc1a39b26073493ca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/mirror_compare2.sh'
2023-11-28T19:55:45.953 INFO:scrape:2 jobs: ['7470277', '7470275']
2023-11-28T19:55:45.953 INFO:scrape:suites intersection: ['conf', 'install/ceph', 'rbd:mirror-compare/{clusters/2-node']
2023-11-28T19:55:45.953 INFO:scrape:suites union: ['conf', 'install/ceph', 'rbd:mirror-compare/{clusters/2-node', 'tasks/krbd_mirror_compare2}', 'tasks/rbd_nbd_mirror_compare2}']
2023-11-28T19:55:45.953 INFO:scrape:
2023-11-28T19:55:45.954 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7470274]  rbd:mirror-compare/{clusters/2-node conf install/ceph tasks/krbd_mirror_compare}
-----------------------------------------------------------------
time:   00:19:19
info:   https://pulpito.ceph.com/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470274/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470274/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1413deba794646638eb60f2648951b89

    Command failed (workunit test rbd/mirror_compare.sh) on smithi119 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c60921579da9a8c8d5d71cddc1a39b26073493ca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=krbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/mirror_compare.sh'

[7470275]  rbd:mirror-compare/{clusters/2-node conf install/ceph tasks/krbd_mirror_compare2}
-----------------------------------------------------------------
time:   00:18:40
info:   https://pulpito.ceph.com/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470275/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470275/
sentry: https://sentry.ceph.com/organizations/ceph/?query=021184363f7f41caa3ffd2417c0e021b

    Command failed (workunit test rbd/mirror_compare2.sh) on smithi112 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c60921579da9a8c8d5d71cddc1a39b26073493ca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=krbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/mirror_compare2.sh'

[7470276]  rbd:mirror-compare/{clusters/2-node conf install/ceph tasks/rbd_nbd_mirror_compare}
-----------------------------------------------------------------
time:   00:20:22
info:   https://pulpito.ceph.com/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470276/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470276/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c7a64a170d0e4b82b0fec0890a964755

    Command failed (workunit test rbd/mirror_compare.sh) on smithi114 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c60921579da9a8c8d5d71cddc1a39b26073493ca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/mirror_compare.sh'

[7470277]  rbd:mirror-compare/{clusters/2-node conf install/ceph tasks/rbd_nbd_mirror_compare2}
-----------------------------------------------------------------
time:   00:18:12
info:   https://pulpito.ceph.com/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470277/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-11-28_19:22:59-rbd:mirror-compare-wip-ajarr-61617-distro-default-smithi/7470277/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db3579e3d98e4175b5cc0e42bd8a1ba5

    Command failed (workunit test rbd/mirror_compare2.sh) on smithi155 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c60921579da9a8c8d5d71cddc1a39b26073493ca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/mirror_compare2.sh'