2024-01-29T22:41:06.831 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-29T22:41:06.831 INFO:teuthology.results:seed: '4100' 2024-01-29T22:47:24.077 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-29T22:47:24.081 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-30T10:48:13.849 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-01-30T10:48:14.060 INFO:scrape:Found 4 jobs 2024-01-30T10:48:14.579 INFO:scrape:Found 3 distinct failure reasons 2024-01-30T10:48:14.580 INFO:scrape:Failure: Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e71f514c30be8dcda7dbbbd9132fbc9729e5332a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh' 2024-01-30T10:48:14.580 INFO:scrape:1 jobs: ['7537471'] 2024-01-30T10:48:14.580 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd}'] 2024-01-30T10:48:14.580 INFO:scrape: 2024-01-30T10:48:14.580 INFO:scrape:Dead: b'2024-01-29T23:02:36.538 INFO:teuthology.orchestra.run.smithi150.stdout: 265292 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 29 23:02 /home/ubuntu/cephtest' 2024-01-30T10:48:14.580 INFO:scrape:1 jobs: ['7537470'] 2024-01-30T10:48:14.580 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd}'] 2024-01-30T10:48:14.581 INFO:scrape: 2024-01-30T10:48:14.581 INFO:scrape:Dead: b'2024-01-30T10:47:35.580 DEBUG:teuthology.exit:Finished running handlers' 2024-01-30T10:48:14.581 INFO:scrape:2 jobs: ['7537469', '7537468'] 2024-01-30T10:48:14.581 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd:mirror/{base/install'] 2024-01-30T10:48:14.581 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rbd:mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd}', 'workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd}'] 2024-01-30T10:48:14.581 INFO:scrape: 2024-01-30T10:48:14.581 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/ failed: 1 dead: 0 running: 3 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7537471] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd} ----------------------------------------------------------------- time: 00:12:37 info: https://pulpito.ceph.com/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7537471/ log: http://qa-proxy.ceph.com/teuthology/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7537471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fe0d63ff5a944598a19faded1c64b5b Command failed (workunit test rbd/compare_mirror_image_alternate_primary.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e71f514c30be8dcda7dbbbd9132fbc9729e5332a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/comp are_mirror_image_alternate_primary.sh' Running ================================================================= [7537468] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd} info: https://pulpito.ceph.com/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7537468/ [7537469] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-nbd} info: https://pulpito.ceph.com/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7537469/ [7537470] rbd:mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-workunit-compare-mirror-image-alternate-primary-krbd} info: https://pulpito.ceph.com/rraja-2024-01-29_22:40:24-rbd:mirror-wip-ajarr-61617-revert-corruption-fixes-distro-default-smithi/7537470/