2023-12-08T14:40:16.851 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-08T14:40:16.852 INFO:teuthology.results:seed: '1629'
2023-12-08T14:44:20.183 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-08T14:44:20.187 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-12-08T15:13:22.453 INFO:teuthology.results:Tests finished! gathering results...
2023-12-08T15:13:22.616 INFO:scrape:Found 4 jobs
2023-12-08T15:13:22.735 INFO:scrape:Found 1 distinct failure reasons
2023-12-08T15:13:22.736 INFO:scrape:Failure: Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi132 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=abd95fb1bf85f683c3d3eb7d8c1b185f1e0e6f70 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=krbd RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh'
2023-12-08T15:13:22.736 INFO:scrape:4 jobs: ['7484106', '7484109', '7484107', '7484108']
2023-12-08T15:13:22.736 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install']
2023-12-08T15:13:22.736 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd}']
2023-12-08T15:13:22.736 INFO:scrape:
2023-12-08T15:13:22.737 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7484106]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd}
-----------------------------------------------------------------
time:   00:17:22
info:   https://pulpito.ceph.com/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484106/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484106/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5155b22e67b7482788de1b18c9cf5f26

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi132 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=abd95fb1bf85f683c3d3eb7d8c1b185f1e0e6f70
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=krbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ge_alternate_primary.sh'

[7484107]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd}
-----------------------------------------------------------------
time:   00:16:37
info:   https://pulpito.ceph.com/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484107/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484107/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d3865e30507448aa406a2962fcf34eb

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi204 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=abd95fb1bf85f683c3d3eb7d8c1b185f1e0e6f70
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ge_alternate_primary.sh'

[7484108]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd}
-----------------------------------------------------------------
time:   00:19:00
info:   https://pulpito.ceph.com/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484108/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484108/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3a48d1c68fd94dc1aa265c879073fbd5

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi111 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=abd95fb1bf85f683c3d3eb7d8c1b185f1e0e6f70
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=krbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ge_alternate_primary.sh'

[7484109]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd}
-----------------------------------------------------------------
time:   00:19:29
info:   https://pulpito.ceph.com/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484109/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2023-12-08_14:39:38-rbd:mirror-main-distro-default-smithi/7484109/
sentry: https://sentry.ceph.com/organizations/ceph/?query=16cdd6af59264f7e9c628f0199b19931

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi097 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=abd95fb1bf85f683c3d3eb7d8c1b185f1e0e6f70
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_DEVICE_TYPE=nbd
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_INSTANCES=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits
    ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt
    u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_ima
    ge_alternate_primary.sh'