2023-10-03T19:55:38.285 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T19:55:38.285 INFO:teuthology.results:seed: '5538' 2023-10-03T19:55:53.841 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T19:55:53.841 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-04T00:04:12.168 INFO:teuthology.results:Tests finished! gathering results... 2023-10-04T00:04:13.400 INFO:scrape:Found 20 jobs 2023-10-04T00:04:13.407 INFO:scrape:Found 1 distinct failure reasons 2023-10-04T00:04:13.408 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-04T00:04:13.408 INFO:scrape:5 jobs: ['7409853', '7409839', '7409840', '7409856', '7409854'] 2023-10-04T00:04:13.408 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'msgr-failures/few', 'openstack}', 'rbd:cli/{base/install', 'workloads/rbd_support_module_recovery}'] 2023-10-04T00:04:13.408 INFO:scrape:suites union: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/defaults', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'pool/none', 'pool/small-cache-pool', 'rbd:cli/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_support_module_recovery}'] 2023-10-04T00:04:13.408 INFO:scrape: 2023-10-04T00:04:13.408 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [7409839] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:41:41 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409839/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409839/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da660bd9b9e44d4487a3252d059bfb26 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi098 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409840] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:14:14 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409840/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=604d34fb4844432cabe9aed055f02d4f Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi169 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409853] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:20:17 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409853/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409853/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61d7de44cd0d4b148bc627570a87fdcf Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi012 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409854] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:56:14 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409854/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5feec76fd4c549f296cd2d71ec4d06c4 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi186 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409856] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:33:52 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409856/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c90b72c9d08e47ab8a11a6562d6583be Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi191 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' Pass ================================================================= [7409838] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:33:41 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409838/ [7409841] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 02:03:41 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409841/ [7409842] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 02:07:56 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409842/ [7409843] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 01:59:53 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409843/ [7409844] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:58:30 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409844/ [7409845] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:31:52 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409845/ [7409846] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:57:49 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409846/ [7409847] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:34:22 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409847/ [7409848] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:35:39 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409848/ [7409849] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:34:20 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409849/ [7409850] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 02:03:48 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409850/ [7409851] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:52:27 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409851/ [7409852] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:39:36 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409852/ [7409855] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:53:53 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409855/ [7409857] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:49:38 info: http://pulpito.ceph.com/rraja-2023-10-03_19:55:32-rbd:cli-wip-ajarr-63009-schedule-reacquire-kickstart-distro-default-smithi/7409857/