2023-10-27T14:25:46.323 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-27T14:25:46.324 INFO:teuthology.results:subset: '111/120000' 2023-10-27T14:25:46.324 INFO:teuthology.results:seed: '4707' 2023-10-27T18:11:13.756 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-27T18:11:13.785 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-27T22:37:34.145 INFO:teuthology.results:Tests finished! gathering results... 2023-10-27T22:37:36.530 INFO:scrape:Found 27 jobs 2023-10-27T22:37:37.272 INFO:scrape:Found 2 distinct failure reasons 2023-10-27T22:37:37.273 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-10-27T22:37:37.273 INFO:scrape:2 jobs: ['7438912', '7438930'] 2023-10-27T22:37:37.274 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-10-27T22:37:37.274 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-10-27T22:37:37.274 INFO:scrape: 2023-10-27T22:37:37.275 INFO:scrape:Failure: Command failed on smithi072 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3048b60-74f4-11ee-8db9-212e2dc638e7 -e sha1=583408857e5d6da3d29992dea1b6a77cbbc9bc3a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-10-27T22:37:37.275 INFO:scrape:1 jobs: ['7438927'] 2023-10-27T22:37:37.276 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default'] 2023-10-27T22:37:37.276 INFO:scrape: 2023-10-27T22:37:37.278 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [7438912] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:49 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438912/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=84c027d3b3b64347ad048a428eb7417e Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7438927] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:57:57 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438927/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fb133d47dda4d6291c3aa8f0fc281ac Command failed on smithi072 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3048b60-74f4-11ee-8db9-212e2dc638e7 -e sha1=583408857e5d6da3d29992dea1b6a77cbbc9bc3a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7438930] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:41 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438930/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438930/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59c74f8552f64430855b4140cd9a99f7 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Pass ================================================================= [7438897] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:25 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438897/ [7438898] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:36:36 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438898/ [7438899] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:04:52 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438899/ [7438901] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:52:23 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438901/ [7438903] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:06 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438903/ [7438905] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:21 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438905/ [7438908] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:00:45 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438908/ [7438910] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:25 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438910/ [7438914] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:22:14 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438914/ [7438916] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:08:35 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438916/ [7438918] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:07:49 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438918/ [7438920] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:34 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438920/ [7438921] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:36 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438921/ [7438922] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:03:25 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438922/ [7438923] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:19:06 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438923/ [7438924] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:03:24 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438924/ [7438925] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:12 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438925/ [7438926] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:56:55 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438926/ [7438928] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:10:30 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438928/ [7438929] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:42 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438929/ [7438931] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:46:59 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438931/ [7438932] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:19:05 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438932/ [7438933] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:23 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438933/ [7438934] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:52:58 info: http://pulpito.ceph.com/teuthology-2023-10-27_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7438934/