2024-02-23T14:33:11.544 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-23T14:33:11.544 INFO:teuthology.results:seed: '3836' 2024-02-23T14:33:24.449 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-23T14:33:24.449 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-23T17:27:37.113 INFO:teuthology.results:Tests finished! gathering results... 2024-02-23T17:27:37.749 INFO:scrape:Found 10 jobs 2024-02-23T17:27:38.837 INFO:scrape:Found 1 distinct failure reasons 2024-02-23T17:27:38.837 INFO:scrape:Failure: Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b7aae5c3c5a1dd24c4cb7ceb499292af00bae680 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh' 2024-02-23T17:27:38.837 INFO:scrape:5 jobs: ['7571979', '7571983', '7571975', '7571977', '7571981'] 2024-02-23T17:27:38.837 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install', 'workloads/compare-mirror-image-alternate-primary-nbd}'] 2024-02-23T17:27:38.837 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/compare-mirror-image-alternate-primary-nbd}'] 2024-02-23T17:27:38.838 INFO:scrape: 2024-02-23T17:27:38.838 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7571975] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-nbd} ----------------------------------------------------------------- time: 00:19:35 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571975/ log: http://qa-proxy.ceph.com/teuthology/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571975/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea0430956674476daa2bae6d6b07f334 Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b7aae5c3c5a1dd24c4cb7ceb499292af00bae680 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp are_mirror_image_alternate_primary.sh' [7571977] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/compare-mirror-image-alternate-primary-nbd} ----------------------------------------------------------------- time: 00:17:09 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571977/ log: http://qa-proxy.ceph.com/teuthology/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571977/ sentry: https://sentry.ceph.com/organizations/ceph/?query=267b6f34b2bc47f0a9ce8d3b6016eaec Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b7aae5c3c5a1dd24c4cb7ceb499292af00bae680 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp are_mirror_image_alternate_primary.sh' [7571979] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} workloads/compare-mirror-image-alternate-primary-nbd} ----------------------------------------------------------------- time: 00:23:31 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571979/ log: http://qa-proxy.ceph.com/teuthology/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571979/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8a57d022b844b41860c59d412cc92d0 Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b7aae5c3c5a1dd24c4cb7ceb499292af00bae680 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp are_mirror_image_alternate_primary.sh' [7571981] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-nbd} ----------------------------------------------------------------- time: 00:20:03 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571981/ log: http://qa-proxy.ceph.com/teuthology/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571981/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d00a9c06f8d041e89e3a4d84c53acd1a Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b7aae5c3c5a1dd24c4cb7ceb499292af00bae680 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp are_mirror_image_alternate_primary.sh' [7571983] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-nbd} ----------------------------------------------------------------- time: 00:20:24 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571983/ log: http://qa-proxy.ceph.com/teuthology/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571983/ sentry: https://sentry.ceph.com/organizations/ceph/?query=791a6fc796004d7aa18d4bd29d8af629 Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b7aae5c3c5a1dd24c4cb7ceb499292af00bae680 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp are_mirror_image_alternate_primary.sh' Pass ================================================================= [7571974] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} workloads/compare-mirror-image-alternate-primary-krbd} time: 02:35:46 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571974/ [7571976] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} workloads/compare-mirror-image-alternate-primary-krbd} time: 02:35:53 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571976/ [7571978] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-krbd} time: 02:41:08 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571978/ [7571980] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-krbd} time: 02:42:07 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571980/ [7571982] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-image-alternate-primary-krbd} time: 02:43:55 info: https://pulpito.ceph.com/rraja-2024-02-23_14:33:05-rbd:mirror-wip-ajarr-revert-librbd-corruption-fixes-feb-22-1-distro-default-smithi/7571982/