2023-10-03T17:10:43.575 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T17:10:43.575 INFO:teuthology.results:seed: '8213' 2023-10-03T17:11:41.743 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T17:11:41.746 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-03T19:28:52.015 INFO:teuthology.results:Tests finished! gathering results... 2023-10-03T19:28:53.262 INFO:scrape:Found 20 jobs 2023-10-03T19:28:53.716 INFO:scrape:Found 2 distinct failure reasons 2023-10-03T19:28:53.716 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi083 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T19:28:53.716 INFO:scrape:3 jobs: ['7409694', '7409690', '7409699'] 2023-10-03T19:28:53.717 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'msgr-failures/few', 'openstack}', 'rbd:cli/{base/install', 'workloads/rbd_support_module_recovery}'] 2023-10-03T19:28:53.717 INFO:scrape:suites union: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/defaults', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'pool/ec-data-pool', 'pool/none', 'pool/replicated-data-pool', 'rbd:cli/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_support_module_recovery}'] 2023-10-03T19:28:53.717 INFO:scrape: 2023-10-03T19:28:53.717 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi037 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T19:28:53.717 INFO:scrape:ceph version 18.0.0-6336-ga7de91bf (a7de91bfa5778a0d559a7e9bc8e35f93456b41ad) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f263a508cf0] 2: PyBytes_FromString() 3: /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so(+0x86044) [0x7f25dbff1044] 4: /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so(+0x2f0df) [0x7f25dbf9a0df] 5: /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so(+0x54b3b) [0x7f25dbfbfb3b] 6: /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so(+0x201e4) [0x7f25dbf8b1e4] 7: /lib64/librbd.so.1(+0x2d29c4) [0x7f25db8089c4] 8: /lib64/librbd.so.1(+0x2d3d78) [0x7f25db809d78] 9: /lib64/librbd.so.1(+0x2d4660) [0x7f25db80a660] 10: /lib64/librbd.so.1(+0x2d4a7d) [0x7f25db80aa7d] 11: /lib64/librbd.so.1(+0x410b65) [0x7f25db946b65] 12: /lib64/librados.so.2(+0xd41ca) [0x7f26388a41ca] 13: /lib64/librados.so.2(+0xc2401) [0x7f2638892401] 14: /lib64/librados.so.2(+0xd8b5f) [0x7f26388a8b5f] 15: /lib64/libstdc++.so.6(+0xc2b13) [0x7f26393eab13] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f263a4fe1ca] 2023-10-03T19:28:53.717 INFO:scrape:1 jobs: ['7409704'] 2023-10-03T19:28:53.717 INFO:scrape:suites: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'pool/small-cache-pool', 'rbd:cli/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd_support_module_recovery}'] 2023-10-03T19:28:53.717 INFO:scrape: 2023-10-03T19:28:53.718 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [7409690] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 01:31:36 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aabfa6783e8e4e5a9628900ae137d4a1 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi031 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409694] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 01:40:16 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409694/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=846c24d33fd14fa18badeae64535019d Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi083 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409699] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 00:34:01 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d3bf0ae55a041efb7d9f3ffc22fb248 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi111 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7409704] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 00:46:41 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409704/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a4164ae89a24477a2058eac8c34df4e Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi037 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' Pass ================================================================= [7409691] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 02:02:57 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409691/ [7409692] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:33:54 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409692/ [7409693] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 02:05:27 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409693/ [7409695] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:31:31 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409695/ [7409696] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-zstd pool/ec-data-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:36:19 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409696/ [7409697] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/replicated-data-pool supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 01:56:27 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409697/ [7409698] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:33:56 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409698/ [7409700] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:34:12 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409700/ [7409701] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/replicated-data-pool supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 01:56:56 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409701/ [7409702] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} time: 01:37:38 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409702/ [7409703] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/none supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:31:50 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409703/ [7409705] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 01:54:27 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409705/ [7409706] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 02:03:25 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409706/ [7409707] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/none supported-random-distro$/{centos_8} workloads/rbd_support_module_recovery} time: 01:31:54 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409707/ [7409708] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} time: 02:05:25 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409708/ [7409709] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-stupid pool/none supported-random-distro$/{ubuntu_20.04} workloads/rbd_support_module_recovery} time: 01:55:27 info: http://pulpito.ceph.com/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409709/