2023-11-06T23:42:44.149 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-06T23:42:44.150 INFO:teuthology.results:subset: '111/120000' 2023-11-06T23:42:44.150 INFO:teuthology.results:seed: '5829' 2023-11-07T03:38:40.714 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-07T03:38:40.740 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-07T07:49:59.925 INFO:teuthology.results:Tests finished! gathering results... 2023-11-07T07:50:02.932 INFO:scrape:Found 27 jobs 2023-11-07T07:50:03.049 INFO:scrape:Found 1 distinct failure reasons 2023-11-07T07:50:03.050 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-11-07T07:50:03.050 INFO:scrape:2 jobs: ['7445550', '7445572'] 2023-11-07T07:50:03.050 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-11-07T07:50:03.051 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-11-07T07:50:03.051 INFO:scrape: 2023-11-07T07:50:03.053 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [7445550] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:57 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445550/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445550/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6e7062bcd364866912dcebf08158122 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7445572] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:51 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445572/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445572/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a91ae7997634c92a221994ad3595a78 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Pass ================================================================= [7445539] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:03 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445539/ [7445540] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:19:14 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445540/ [7445541] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:24:25 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445541/ [7445542] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:57:54 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445542/ [7445543] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:37 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445543/ [7445544] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:49 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445544/ [7445546] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:02:52 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445546/ [7445548] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:32 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445548/ [7445553] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:18:50 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445553/ [7445554] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:11:15 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445554/ [7445556] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:09:59 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445556/ [7445559] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:22 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445559/ [7445560] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:29 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445560/ [7445562] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:05:49 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445562/ [7445565] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:17:24 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445565/ [7445566] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:14:58 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445566/ [7445567] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:03 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445567/ [7445568] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:17:04 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445568/ [7445569] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:42 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445569/ [7445570] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:16:52 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445570/ [7445571] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:17 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445571/ [7445573] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:45:52 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445573/ [7445574] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:01:52 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445574/ [7445575] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:21 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445575/ [7445576] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:42:15 info: https://pulpito.ceph.com/teuthology-2023-11-03_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7445576/