2023-09-30T23:36:40.027 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T23:36:40.028 INFO:teuthology.results:subset: '111/120000' 2023-09-30T23:36:40.028 INFO:teuthology.results:seed: '3618' 2023-10-01T01:08:12.337 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-01T01:08:12.363 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-01T05:42:33.109 INFO:teuthology.results:Tests finished! gathering results... 2023-10-01T05:42:35.804 INFO:scrape:Found 27 jobs 2023-10-01T05:42:35.968 INFO:scrape:Found 1 distinct failure reasons 2023-10-01T05:42:35.968 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-10-01T05:42:35.968 INFO:scrape:2 jobs: ['7406359', '7406338'] 2023-10-01T05:42:35.968 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-10-01T05:42:35.969 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-10-01T05:42:35.969 INFO:scrape: 2023-10-01T05:42:35.971 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [7406338] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:48 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406338/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc1a8699fbb14536b97616ecc9c8048e Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7406359] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:41 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406359/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01f007aa500346c38b9da91c5ded38ae Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Pass ================================================================= [7406326] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:39 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406326/ [7406327] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:02:17 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406327/ [7406328] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:30:20 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406328/ [7406329] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:59:42 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406329/ [7406330] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:41 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406330/ [7406332] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:46 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406332/ [7406334] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:28:09 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406334/ [7406336] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:51 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406336/ [7406340] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:40:15 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406340/ [7406342] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:32:11 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406342/ [7406344] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:00:49 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406344/ [7406346] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:46 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406346/ [7406348] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:18 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406348/ [7406350] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:15:06 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406350/ [7406352] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:17:31 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406352/ [7406353] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:36:49 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406353/ [7406354] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:28 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406354/ [7406355] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:05:27 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406355/ [7406356] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:01 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406356/ [7406357] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:15:09 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406357/ [7406358] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:16 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406358/ [7406360] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:41:37 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406360/ [7406361] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:33:45 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406361/ [7406362] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:51 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406362/ [7406363] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:51:24 info: http://pulpito.ceph.com/teuthology-2023-09-29_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7406363/