2023-10-19T02:22:22.285 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-19T02:22:22.286 INFO:teuthology.results:seed: '4889' 2023-10-19T04:35:33.360 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-19T04:35:33.391 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-19T07:29:46.423 INFO:teuthology.results:Tests finished! gathering results... 2023-10-19T07:29:48.193 INFO:scrape:Found 20 jobs 2023-10-19T07:29:49.504 INFO:scrape:Found 1 distinct failure reasons 2023-10-19T07:29:49.504 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b68191d8fe638ade2478310e80bbeb113cd424bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-19T07:29:49.505 INFO:scrape:4 jobs: ['7431962', '7431952', '7431956', '7431948'] 2023-10-19T07:29:49.505 INFO:scrape:suites intersection: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'msgr-failures/few', 'openstack}', 'rbd:cli/{base/install', 'workloads/rbd_support_module_recovery}'] 2023-10-19T07:29:49.506 INFO:scrape:suites union: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'data-pool/ec', 'data-pool/replicated', 'features/defaults', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd:cli/{base/install', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_support_module_recovery}'] 2023-10-19T07:29:49.506 INFO:scrape: 2023-10-19T07:29:49.507 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [7431948] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:07:17 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431948/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b83906fb2f74f4295b1693ab1bc4a9a Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b68191d8fe638ade2478310e80bbeb113cd424bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7431952] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:49:14 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431952/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8224cabee644b25adb3732c9efa6110 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b68191d8fe638ade2478310e80bbeb113cd424bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7431956] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:44:29 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431956/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431956/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00d3990d3e2840c4ac2226e3d16e7119 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b68191d8fe638ade2478310e80bbeb113cd424bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' [7431962] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/defaults msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} ----------------------------------------------------------------- time: 02:37:22 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431962/ log: http://qa-proxy.ceph.com/teuthology/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431962/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5b37b471def44c3930d7a085d7948b5 Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b68191d8fe638ade2478310e80bbeb113cd424bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' Pass ================================================================= [7431947] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_migration} time: 00:26:36 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431947/ [7431949] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export} time: 00:12:21 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431949/ [7431950] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/defaults msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_luks_encryption} time: 00:14:19 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431950/ [7431951] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_migration} time: 00:29:51 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431951/ [7431953] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/defaults msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} time: 00:31:08 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431953/ [7431954] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export} time: 00:12:23 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431954/ [7431955] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_migration} time: 00:29:49 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431955/ [7431957] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/defaults msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} time: 00:30:01 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431957/ [7431958] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_groups} time: 00:15:32 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431958/ [7431959] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export} time: 00:12:21 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431959/ [7431960] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_migration} time: 00:26:21 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431960/ [7431961] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} workloads/rbd_support_module_recovery} time: 01:53:53 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431961/ [7431963] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic} time: 00:32:00 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431963/ [7431964] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_groups} time: 00:15:49 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431964/ [7431965] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/defaults msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_import_export} time: 00:12:39 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431965/ [7431966] rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_luks_encryption} time: 00:12:02 info: http://pulpito.ceph.com/rraja-2023-10-19_02:20:01-rbd:cli-ajarr-wip-more-debug-62994-oct-18-distro-default-smithi/7431966/