2023-07-14T22:10:54.295 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-14T22:10:54.296 INFO:teuthology.results:subset: '111/120000' 2023-07-14T22:10:54.296 INFO:teuthology.results:seed: '4171' 2023-07-14T23:11:53.249 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-14T23:11:53.280 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-15T02:28:08.856 INFO:teuthology.results:Tests finished! gathering results... 2023-07-15T02:28:11.506 INFO:scrape:Found 27 jobs 2023-07-15T02:28:19.941 INFO:scrape:Found 3 distinct failure reasons 2023-07-15T02:28:19.942 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-07-15T02:28:19.943 INFO:scrape:16 jobs: ['7337921', '7337929', '7337930', '7337915', '7337919', '7337894', '7337911', '7337900', '7337898', '7337926', '7337907', '7337924', '7337899', '7337913', '7337923', '7337931'] 2023-07-15T02:28:19.943 INFO:scrape:suites intersection: [] 2023-07-15T02:28:19.944 INFO:scrape:suites union: ['0-roles', '0-start', '1-ceph-install/octopus', '1-start', '1-tasks', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/snaps-few-objects', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api}}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-07-15T02:28:19.944 INFO:scrape: 2023-07-15T02:28:19.944 INFO:scrape:Failure: Command failed on smithi042 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8648228c-2298-11ee-9b33-001a4aab830c -e sha1=8da5fd3a8b53ca9afd456e98a74fb2fc7639aae6 -- bash -c 'ceph orch ps'" 2023-07-15T02:28:19.945 INFO:scrape:2 jobs: ['7337909', '7337901'] 2023-07-15T02:28:19.945 INFO:scrape:suites intersection: [] 2023-07-15T02:28:19.945 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default'] 2023-07-15T02:28:19.946 INFO:scrape: 2023-07-15T02:28:19.946 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-07-15T02:28:19.946 INFO:scrape:2 jobs: ['7337927', '7337902'] 2023-07-15T02:28:19.946 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-07-15T02:28:19.947 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-07-15T02:28:19.947 INFO:scrape: 2023-07-15T02:28:19.950 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 20 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7337894] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:44:42 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337894/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d89b642e948c4bc78ba0f8ecbe67e78f Command failed (workunit test cls/test_cls_rbd.sh) on smithi177 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337898] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:38:47 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337898/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337898/ sentry: https://sentry.ceph.com/organizations/ceph/?query=721fe31cd63f47879e5ef20122be41ea Command failed (workunit test cls/test_cls_rbd.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337899] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:05 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337899/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b26c28c9122464c988123c4e62414d1 Command failed (workunit test cls/test_cls_rbd.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337900] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:31:36 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337900/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc21e0697e3f4deb90d82002ae5ad5dc Command failed (workunit test cls/test_cls_rbd.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337901] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 01:01:17 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337901/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4577270474664cf08bdbb1b2f1c48f04 Command failed on smithi107 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57275a0e-2298-11ee-9b33-001a4aab830c -e sha1=8da5fd3a8b53ca9afd456e98a74fb2fc7639aae6 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7337902] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:08:13 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337902/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7126f67ff5e0404b90c0f6037606e0dd Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7337907] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:13:07 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337907/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7885f970aca448aaa615789ce962fe4a Command failed (workunit test cls/test_cls_rbd.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337909] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:57:11 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337909/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337909/ sentry: https://sentry.ceph.com/organizations/ceph/?query=749f503be7664398bbc64bed9211c9d7 Command failed on smithi042 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8648228c-2298-11ee-9b33-001a4aab830c -e sha1=8da5fd3a8b53ca9afd456e98a74fb2fc7639aae6 -- bash -c 'ceph orch ps'" [7337911] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:38:48 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337911/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ece4e9a87f74de2adecf3595d4e5953 Command failed (workunit test cls/test_cls_rbd.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337913] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:42:15 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337913/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3b96779519e47bb80b37c48f77af494 Command failed (workunit test cls/test_cls_rbd.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337915] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:39:53 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337915/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337915/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b484ffab63574bd7a0894d14cf0979f3 Command failed (workunit test cls/test_cls_rbd.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337919] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:36:37 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337919/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337919/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9955c0524bd946cfaf2ce9e69cb92bb7 Command failed (workunit test cls/test_cls_rbd.sh) on smithi182 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337921] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:38:09 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337921/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=44e9ba89ae12490d8a7491b967039ab9 Command failed (workunit test cls/test_cls_rbd.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337923] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:46:19 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337923/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337923/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6d741941b6e41008a8aa5e68778f885 Command failed (workunit test cls/test_cls_rbd.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337924] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:51 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337924/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1dfaa343d64c4c698a52ef18171ff4c1 Command failed (workunit test cls/test_cls_rbd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337926] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:33:25 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337926/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ab5b798924b4f5cbb0694d8e84d4d81 Command failed (workunit test cls/test_cls_rbd.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337927] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:55 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337927/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40c1c7d84d564a11b25638a1215d09c6 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7337929] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:08:30 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337929/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1b8a3a781934a0187fc41db289109f6 Command failed (workunit test cls/test_cls_rbd.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337930] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:34:56 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337930/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337930/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f75aa02ed62847ed919df16d31d5aa96 Command failed (workunit test cls/test_cls_rbd.sh) on smithi177 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7337931] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:38:43 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337931/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337931/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7460e08417304dad8bc22753fad36ecc Command failed (workunit test cls/test_cls_rbd.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' Pass ================================================================= [7337895] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:21:25 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337895/ [7337896] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:36:43 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337896/ [7337897] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:03:18 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337897/ [7337905] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:53:57 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337905/ [7337917] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:25:46 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337917/ [7337925] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:10:45 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337925/ [7337928] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:44:59 info: http://pulpito.ceph.com/teuthology-2023-07-14_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7337928/