2023-05-26T15:03:21.155 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b 2023-05-26T15:03:21.156 INFO:teuthology.results:subset: '111/120000' 2023-05-26T15:03:21.156 INFO:teuthology.results:seed: '7499' 2023-05-26T16:13:36.613 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b 2023-05-26T16:13:36.644 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-26T19:35:51.622 INFO:teuthology.results:Tests finished! gathering results... 2023-05-26T19:35:54.246 INFO:scrape:Found 27 jobs 2023-05-26T19:36:01.420 INFO:scrape:Found 2 distinct failure reasons 2023-05-26T19:36:01.421 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-05-26T19:36:01.421 INFO:scrape:18 jobs: ['7287154', '7287153', '7287151', '7287178', '7287162', '7287166', '7287152', '7287177', '7287180', '7287168', '7287156', '7287183', '7287185', '7287170', '7287184', '7287148', '7287176', '7287150'] 2023-05-26T19:36:01.422 INFO:scrape:suites intersection: [] 2023-05-26T19:36:01.422 INFO:scrape:suites union: ['0-roles', '0-start', '1-ceph-install/octopus', '1-start', '1-tasks', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/snaps-few-objects', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api}}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-05-26T19:36:01.422 INFO:scrape: 2023-05-26T19:36:01.423 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-05-26T19:36:01.423 INFO:scrape:2 jobs: ['7287158', '7287181'] 2023-05-26T19:36:01.423 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-05-26T19:36:01.424 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-05-26T19:36:01.424 INFO:scrape: 2023-05-26T19:36:01.427 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 20 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7287148] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:39:30 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287148/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42ddcb7771bd462a9a765cde3991c924 Command failed (workunit test cls/test_cls_rbd.sh) on smithi072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287150] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:31:47 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287150/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb5200c10b34434590a10bf2e4475064 Command failed (workunit test cls/test_cls_rbd.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287151] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:36:13 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287151/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c208dfe8405b43f6881f87085ac22690 Command failed (workunit test cls/test_cls_rbd.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287152] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:39:54 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287152/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=12ff9e25092b43e297fb0078803de51c Command failed (workunit test cls/test_cls_rbd.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287153] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:35:42 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287153/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5785a21da52f4a14bc0abbcfbf7ae723 Command failed (workunit test cls/test_cls_rbd.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287154] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:47:33 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287154/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adff0da9f0334934a4fe9194d2dafbeb Command failed (workunit test cls/test_cls_rbd.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287156] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:33:33 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287156/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287156/ sentry: https://sentry.ceph.com/organizations/ceph/?query=661894ffb537436898155d1258bfe7a1 Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287158] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:33 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287158/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287158/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b79c5dd87614641a87f9703101c8e30 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7287162] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:06:12 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287162/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287162/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f29729bd2d6b4c1484a30fe96e4d010d Command failed (workunit test cls/test_cls_rbd.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287166] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:00 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287166/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287166/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7abb7ad6a1814dafad7ddd6dfa9ea9bb Command failed (workunit test cls/test_cls_rbd.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287168] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:42:09 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287168/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287168/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b8a02ba0c094fc1881d8a05880802f8 Command failed (workunit test cls/test_cls_rbd.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287170] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:38:27 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287170/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287170/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5d3bf2ab79945c6bb9e770170b77653 Command failed (workunit test cls/test_cls_rbd.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287176] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:41:51 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287176/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fcbc40cdb1f44e6a9edcdab22fa84de Command failed (workunit test cls/test_cls_rbd.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287177] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:50:43 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287177/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba924fe6fda94cb68bcbfdaa66d794ad Command failed (workunit test cls/test_cls_rbd.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287178] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:26 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287178/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287178/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c7d02ad54e947e18384ab8176ef43ed Command failed (workunit test cls/test_cls_rbd.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287180] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:07 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ae4ded9663e4fc882e13207284ed1fa Command failed (workunit test cls/test_cls_rbd.sh) on smithi053 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287181] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:42 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287181/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287181/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34ef7bbbdc894672b55bdfe2f57eb6b7 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7287183] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:16:09 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287183/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287183/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5e1f985ee4c451fbb3949672466c08a Command failed (workunit test cls/test_cls_rbd.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287184] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:32:26 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287184/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287184/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7841e6b91a6e4a01a7b4929045f77045 Command failed (workunit test cls/test_cls_rbd.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7287185] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:39:29 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287185/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79dda4dc0b814a9184fb0622a4013b82 Command failed (workunit test cls/test_cls_rbd.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' Pass ================================================================= [7287149] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:22:02 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287149/ [7287160] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:43:30 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287160/ [7287164] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:00:11 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287164/ [7287172] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:14:06 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287172/ [7287174] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:38:45 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287174/ [7287179] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:45:58 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287179/ [7287182] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:43:19 info: http://pulpito.ceph.com/teuthology-2023-05-26_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7287182/