2023-08-18T14:23:23.607 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-18T14:23:23.608 INFO:teuthology.results:subset: '111/120000' 2023-08-18T14:23:23.609 INFO:teuthology.results:seed: '5381' 2023-08-18T14:24:18.630 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-18T14:24:18.631 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-18T19:08:37.052 INFO:teuthology.results:Tests finished! gathering results... 2023-08-18T19:08:39.872 INFO:scrape:Found 27 jobs 2023-08-18T19:08:40.701 INFO:scrape:Found 2 distinct failure reasons 2023-08-18T19:08:40.702 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-08-18T19:08:40.702 INFO:scrape:2 jobs: ['7372402', '7372388'] 2023-08-18T19:08:40.702 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-08-18T19:08:40.703 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-08-18T19:08:40.703 INFO:scrape: 2023-08-18T19:08:40.703 INFO:scrape:Failure: Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f34f6e86-3dd6-11ee-9b39-001a4aab830c -e sha1=711bb867dd9d11267a81332c76ab7285b4ab79c5 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-08-18T19:08:40.704 INFO:scrape:2 jobs: ['7372393', '7372386'] 2023-08-18T19:08:40.704 INFO:scrape:suites intersection: [] 2023-08-18T19:08:40.704 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8', 'workload/{ec-rados-default'] 2023-08-18T19:08:40.705 INFO:scrape: 2023-08-18T19:08:40.707 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 23 Fail ================================================================= [7372386] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:00:25 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372386/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5d5709622bf49da9b1ab44e22324864 Command failed on smithi146 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aca8de0-3dd7-11ee-9b39-001a4aab830c -e sha1=711bb867dd9d11267a81332c76ab7285b4ab79c5 -- bash -c 'ceph orch ps'" [7372388] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:08:08 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372388/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a10092cffa55450b82f48ff105ee84bf Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7372393] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 01:02:54 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372393/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372393/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a7f7abf6cc4435cad741ae2bb5b7fd1 Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f34f6e86-3dd6-11ee-9b39-001a4aab830c -e sha1=711bb867dd9d11267a81332c76ab7285b4ab79c5 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7372402] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:08:59 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372402/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b94919db2a704ca99c44d4548346e3ad Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Pass ================================================================= [7372380] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:56 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372380/ [7372381] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:52:53 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372381/ [7372382] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:37:52 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372382/ [7372383] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:54:29 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372383/ [7372384] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:52 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372384/ [7372385] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:11:05 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372385/ [7372387] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:51 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372387/ [7372389] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:50:56 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372389/ [7372390] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:05:06 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372390/ [7372391] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:57:51 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372391/ [7372392] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:19 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372392/ [7372394] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:16:01 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372394/ [7372395] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:22:50 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372395/ [7372396] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:48:43 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372396/ [7372397] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:45 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372397/ [7372398] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:10:58 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372398/ [7372399] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:36 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372399/ [7372400] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:09:03 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372400/ [7372401] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:36 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372401/ [7372403] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:46:40 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372403/ [7372404] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:27:31 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372404/ [7372405] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:33 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372405/ [7372406] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:13:18 info: http://pulpito.ceph.com/teuthology-2023-08-18_14:22:09-upgrade:octopus-x-quincy-distro-default-smithi/7372406/