2023-11-10T22:06:39.540 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-10T22:06:39.541 INFO:teuthology.results:subset: '111/120000' 2023-11-10T22:06:39.541 INFO:teuthology.results:seed: '5033' 2023-11-10T22:11:39.029 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-10T22:11:39.034 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-11T04:16:07.780 INFO:teuthology.results:Tests finished! gathering results... 2023-11-11T04:16:08.536 INFO:scrape:Found 8 jobs 2023-11-11T04:16:09.964 INFO:scrape:Found 2 distinct failure reasons 2023-11-11T04:16:09.965 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_ha.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_ha.sh' 2023-11-11T04:16:09.965 INFO:scrape:1 jobs: ['7455570'] 2023-11-11T04:16:09.965 INFO:scrape:suites: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-ha-workunit}'] 2023-11-11T04:16:09.966 INFO:scrape: 2023-11-11T04:16:09.966 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi136 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=journal MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-11-11T04:16:09.967 INFO:scrape:5 jobs: ['7455572', '7455566', '7455568', '7455565', '7455569'] 2023-11-11T04:16:09.967 INFO:scrape:suites intersection: ['cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'openstack}'] 2023-11-11T04:16:09.968 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-stress-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}'] 2023-11-11T04:16:09.968 INFO:scrape: 2023-11-11T04:16:09.969 INFO:teuthology.results:Sending results to aemerson@redhat.com: Test Run: aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7455565] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-journal-stress-workunit} ----------------------------------------------------------------- time: 00:12:29 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455565/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f1fa00bee08416e8df212b5cd03d316 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi153 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=journal MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _stress.sh' [7455566] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:15:11 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455566/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba7bfba23c5f4ebea02f9beb681f68e0 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi121 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_stress.sh' [7455568] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} ----------------------------------------------------------------- time: 00:12:56 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455568/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455568/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9936f0fa3cb4cd89293bfdf28eb78ee Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi148 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _stress.sh' [7455569] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} ----------------------------------------------------------------- time: 00:12:44 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455569/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455569/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35365e35fa1b4c09af358197a943c39e Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi195 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' [7455570] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:38:19 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455570/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455570/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5b0c75e04fd4c129dc2ea78f9855dd1 Command failed (workunit test rbd/rbd_mirror_ha.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_ha.sh' [7455572] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-journal-stress-workunit} ----------------------------------------------------------------- time: 00:15:40 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455572/ log: http://qa-proxy.ceph.com/teuthology/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455572/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bf6d0fb00f841d9a6e83c704eba1e3e Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi136 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=journal MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /ho me/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror _stress.sh' Pass ================================================================= [7455567] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults} time: 00:52:18 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455567/ [7455571] rbd/qemu/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/qemu_xfstests} time: 05:54:53 info: https://pulpito.ceph.com/aemerson-2023-11-10_22:02:58-rbd-main-distro-default-smithi/7455571/