2023-09-23T06:39:43.248 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-23T06:39:43.249 INFO:teuthology.results:seed: '2241' 2023-09-23T07:01:03.538 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-23T07:01:03.563 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-23T10:55:19.824 INFO:teuthology.results:Tests finished! gathering results... 2023-09-23T10:55:21.282 INFO:scrape:Found 20 jobs 2023-09-23T10:55:23.386 INFO:scrape:Found 2 distinct failure reasons 2023-09-23T10:55:23.386 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-23T10:55:23.386 INFO:scrape:4 jobs: ['7401657', '7401649', '7401660', '7401648'] 2023-09-23T10:55:23.387 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'msgr-failures/few', 'openstack}', 'rbd:cli/{base/install', 'workloads/rbd_support_module_recovery}'] 2023-09-23T10:55:23.387 INFO:scrape:suites union: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/defaults', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'pool/ec-data-pool', 'pool/none', 'pool/replicated-data-pool', 'pool/small-cache-pool', 'rbd:cli/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_support_module_recovery}'] 2023-09-23T10:55:23.387 INFO:scrape: 2023-09-23T10:55:23.387 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-09-23T10:55:23.387 INFO:scrape:3 jobs: ['7401645', '7401646', '7401661'] 2023-09-23T10:55:23.387 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/layering', 'msgr-failures/few', 'openstack}', 'rbd:cli/{base/install', 'workloads/rbd_support_module_recovery}'] 2023-09-23T10:55:23.387 INFO:scrape:suites union: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'pool/none', 'pool/small-cache-pool', 'rbd:cli/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_support_module_recovery}'] 2023-09-23T10:55:23.387 INFO:scrape: 2023-09-23T10:55:23.388 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [7401645] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:49:56 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401645/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401645/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d218ce51100345c59803be3b22133e6d Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi154 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7401646] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:27:15 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401646/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401646/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92c56ec1978c491ea4235534830c8c35 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi187 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7401648] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:06:13 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401648/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401648/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0353baec4204edba4236cc6bcfd410c Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7401649] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 01:50:05 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401649/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5731a07b317e40dc8607ec4c9d013822 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7401657] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:22:05 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401657/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79852ebf7a014a898de16363d542d5c1 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7401660] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:38:15 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401660/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401660/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79fa5b56da624f3eab4062dd84fff27b Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7401661] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 03:43:46 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401661/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401661/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4266d184079643cdb5f523bd09c59c19 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi055 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' Pass ================================================================= [7401643] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:31:23 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401643/ [7401644] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:32:06 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401644/ [7401647] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:37:11 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401647/ [7401650] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:49:31 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401650/ [7401651] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:56:06 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401651/ [7401652] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:48:01 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401652/ [7401653] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:53:58 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401653/ [7401654] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:29:22 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401654/ [7401655] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:32:39 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401655/ [7401656] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:31:16 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401656/ [7401658] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:50:49 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401658/ [7401659] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 02:17:12 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401659/ [7401662] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/none supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 01:57:36 info: http://pulpito.ceph.com/rraja-2023-09-23_06:37:41-rbd:cli-wip-62891-distro-default-smithi/7401662/