2023-08-09T00:40:19.807 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-09T00:40:19.808 INFO:teuthology.results:subset: '111/120000' 2023-08-09T00:40:19.808 INFO:teuthology.results:seed: '9518' 2023-08-09T16:32:05.260 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-09T16:32:05.323 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-09T19:46:20.005 INFO:teuthology.results:Tests finished! gathering results... 2023-08-09T19:46:22.543 INFO:scrape:Found 27 jobs 2023-08-09T19:46:29.574 INFO:scrape:Found 2 distinct failure reasons 2023-08-09T19:46:29.575 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-08-09T19:46:29.575 INFO:scrape:18 jobs: ['7360360', '7360366', '7360350', '7360342', '7360340', '7360372', '7360371', '7360346', '7360364', '7360348', '7360335', '7360345', '7360339', '7360370', '7360358', '7360362', '7360352', '7360337'] 2023-08-09T19:46:29.576 INFO:scrape:suites intersection: [] 2023-08-09T19:46:29.576 INFO:scrape:suites union: ['0-roles', '0-start', '1-ceph-install/octopus', '1-start', '1-tasks', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api}}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default'] 2023-08-09T19:46:29.577 INFO:scrape: 2023-08-09T19:46:29.577 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-08-09T19:46:29.577 INFO:scrape:2 jobs: ['7360368', '7360343'] 2023-08-09T19:46:29.578 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-08-09T19:46:29.578 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-08-09T19:46:29.579 INFO:scrape: 2023-08-09T19:46:29.582 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 20 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7360335] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:42:29 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360335/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db4cd93ff95749a2bd665cd3de88c8a3 Command failed (workunit test cls/test_cls_rbd.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360337] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:28:48 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360337/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4137dfa1e0347ebb2eaf27503368cb3 Command failed (workunit test cls/test_cls_rbd.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360339] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:42:53 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360339/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360339/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81a6d612c59847b598321656dd3d0c93 Command failed (workunit test cls/test_cls_rbd.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360340] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:52 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360340/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=524d0831d8dd4b7b8207b2774e701c61 Command failed (workunit test cls/test_cls_rbd.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360342] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:39:39 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360342/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e5a3b16d53a4ae7a59d0929e27a4565 Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360343] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:08:08 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360343/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=32180de463484fd0ab06667562bbc257 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7360345] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 02:50:43 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360345/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360345/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e941e14e7c9496aa6f6398c7184b70c Command failed (workunit test cls/test_cls_rbd.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360346] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:34:02 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360346/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=426d1682fe17454c9ae2eacc90aa1d31 Command failed (workunit test cls/test_cls_rbd.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360348] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:33:18 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360348/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40229b5229194090861ae8a2cd06aa42 Command failed (workunit test cls/test_cls_rbd.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360350] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:17 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360350/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360350/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc066eab8bbd4b7a849a86ed54dd4d8a Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360352] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:48:52 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360352/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e3490348efd420ab031d8d43b911302 Command failed (workunit test cls/test_cls_rbd.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360358] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:40:09 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360358/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de0ec087291a4548a89570b971ffc3bb Command failed (workunit test cls/test_cls_rbd.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360360] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:48:00 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360360/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fee482f974346c1aedebd3a052fce28 Command failed (workunit test cls/test_cls_rbd.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360362] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:11 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360362/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c040b1a5d2a45b381f499e09403313f Command failed (workunit test cls/test_cls_rbd.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360364] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:31:50 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360364/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0b0b1ea3338424db6ea211a3fde4417 Command failed (workunit test cls/test_cls_rbd.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360366] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:33:48 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360366/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=156028c5775142e5868a5ea56a1f2f93 Command failed (workunit test cls/test_cls_rbd.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360368] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:08:02 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360368/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360368/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce676b49691d4465928db6690e5e60e8 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7360370] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:03:50 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360370/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7aafce4eebe43bcad0e2ec750707a3b Command failed (workunit test cls/test_cls_rbd.sh) on smithi203 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360371] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:20 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360371/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0544bf447a3f4e1bb77fb077b51b999d Command failed (workunit test cls/test_cls_rbd.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7360372] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:32:47 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360372/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33a2d76aa7a44fe78d8e1e02a4240953 Command failed (workunit test cls/test_cls_rbd.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' Pass ================================================================= [7360336] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:35:47 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360336/ [7360338] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:55:14 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360338/ [7360341] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:13:54 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360341/ [7360344] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:04:37 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360344/ [7360354] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:12:37 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360354/ [7360356] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:50:56 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360356/ [7360369] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:44:36 info: http://pulpito.ceph.com/teuthology-2023-08-04_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7360369/