2023-09-22T15:52:16.334 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-22T15:52:16.335 INFO:teuthology.results:subset: '111/120000' 2023-09-22T15:52:16.335 INFO:teuthology.results:seed: '3995' 2023-09-22T19:46:12.127 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-22T19:46:12.173 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-22T22:55:25.970 INFO:teuthology.results:Tests finished! gathering results... 2023-09-22T22:55:28.205 INFO:scrape:Found 27 jobs 2023-09-22T22:55:28.686 INFO:scrape:Found 2 distinct failure reasons 2023-09-22T22:55:28.687 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-09-22T22:55:28.687 INFO:scrape:2 jobs: ['7401095', '7401113'] 2023-09-22T22:55:28.687 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-09-22T22:55:28.687 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-09-22T22:55:28.687 INFO:scrape: 2023-09-22T22:55:28.687 INFO:scrape:Failure: Command failed on smithi082 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1c0c70c-5978-11ee-8db4-212e2dc638e7 -e sha1=d1b37cdb82d062f37148088f5ab44d9476d1bd39 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-09-22T22:55:28.688 INFO:scrape:1 jobs: ['7401116'] 2023-09-22T22:55:28.688 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'workload/{ec-rados-default'] 2023-09-22T22:55:28.688 INFO:scrape: 2023-09-22T22:55:28.690 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [7401095] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:39 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401095/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401095/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27dbd37fe72f4f1d9c7be4b9d59774b4 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7401113] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:45 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401113/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401113/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d61a6670270043459101462a5af26f4a Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7401116] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:54:14 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401116/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401116/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d1e7adb19524a3a9da179b3f0bc4e5f Command failed on smithi082 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1c0c70c-5978-11ee-8db4-212e2dc638e7 -e sha1=d1b37cdb82d062f37148088f5ab44d9476d1bd39 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' Pass ================================================================= [7401080] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:04 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401080/ [7401081] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:26:01 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401081/ [7401083] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:36:33 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401083/ [7401085] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:50:37 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401085/ [7401087] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:30 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401087/ [7401089] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:00 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401089/ [7401091] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 01:59:01 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401091/ [7401093] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:29 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401093/ [7401097] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:29:47 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401097/ [7401099] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:20:26 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401099/ [7401101] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:01:11 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401101/ [7401103] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:37 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401103/ [7401104] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:45 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401104/ [7401105] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:24:59 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401105/ [7401106] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:14:41 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401106/ [7401107] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:34:52 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401107/ [7401108] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:31 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401108/ [7401109] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:03:08 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401109/ [7401110] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:11:30 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401110/ [7401111] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:09:22 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401111/ [7401112] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:09 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401112/ [7401114] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:41:58 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401114/ [7401115] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:27:52 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401115/ [7401117] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:52:21 info: http://pulpito.ceph.com/teuthology-2023-09-22_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7401117/