2024-02-09T16:20:51.528 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-09T16:20:51.528 INFO:teuthology.results:seed: '2453'
2024-02-09T16:40:32.100 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-09T16:40:32.103 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-09T17:10:34.475 INFO:teuthology.results:Tests finished! gathering results...
2024-02-09T17:10:34.654 INFO:scrape:Found 4 jobs
2024-02-09T17:10:34.806 INFO:scrape:Found 1 distinct failure reasons
2024-02-09T17:10:34.806 INFO:scrape:Failure: Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36e841a09f908a3792b02bb243f59904990e39d6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh'
2024-02-09T17:10:34.806 INFO:scrape:4 jobs: ['7554092', '7554093', '7554090', '7554091']
2024-02-09T17:10:34.806 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}']
2024-02-09T17:10:34.806 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/compare-mirror-image-alternate-primary-krbd}', 'workloads/compare-mirror-image-alternate-primary-nbd}']
2024-02-09T17:10:34.807 INFO:scrape:
2024-02-09T17:10:34.807 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7554090]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-krbd}
-----------------------------------------------------------------
time:   00:20:14
info:   https://pulpito.ceph.com/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554090/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554090/
sentry: https://sentry.ceph.com/organizations/ceph/?query=df83df16343b4c78a286f2dd280e919d

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi191 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=36e841a09f908a3792b02bb243f59904990e39d6
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp
    are_mirror_image_alternate_primary.sh'

[7554091]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-krbd}
-----------------------------------------------------------------
time:   00:19:46
info:   https://pulpito.ceph.com/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554091/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554091/
sentry: https://sentry.ceph.com/organizations/ceph/?query=274ea58526654de491e1bb047dddcb18

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi179 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=36e841a09f908a3792b02bb243f59904990e39d6
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp
    are_mirror_image_alternate_primary.sh'

[7554092]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-krbd}
-----------------------------------------------------------------
time:   00:19:52
info:   https://pulpito.ceph.com/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554092/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554092/
sentry: https://sentry.ceph.com/organizations/ceph/?query=be8eee11883b46649d3d94a031f66efd

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi163 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=36e841a09f908a3792b02bb243f59904990e39d6
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp
    are_mirror_image_alternate_primary.sh'

[7554093]  rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-nbd}
-----------------------------------------------------------------
time:   00:19:53
info:   https://pulpito.ceph.com/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554093/
log:    http://qa-proxy.ceph.com/teuthology/rraja-2024-02-09_16:01:40-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7554093/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aeb7f602087143f78b75c8bb796f7c82

    Command failed (workunit test
    rbd/compare_mirror_image_alternate_primary.sh) on smithi104 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=36e841a09f908a3792b02bb243f59904990e39d6
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp
    are_mirror_image_alternate_primary.sh'