2023-10-21T20:23:53.478 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T20:23:53.479 INFO:teuthology.results:subset: '111/120000' 2023-10-21T20:23:53.479 INFO:teuthology.results:seed: '4981' 2023-10-21T21:15:47.979 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T21:15:48.012 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-22T01:42:07.273 INFO:teuthology.results:Tests finished! gathering results... 2023-10-22T01:42:09.833 INFO:scrape:Found 27 jobs 2023-10-22T01:42:10.256 INFO:scrape:Found 2 distinct failure reasons 2023-10-22T01:42:10.257 INFO:scrape:Failure: Command failed on smithi008 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb33f480-7059-11ee-8db6-212e2dc638e7 -e sha1=33809aa16833daf9cbc6787158e533522f8e6233 -- bash -c 'ceph orch ps'" 2023-10-22T01:42:10.257 INFO:scrape:1 jobs: ['7433044'] 2023-10-22T01:42:10.257 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8'] 2023-10-22T01:42:10.258 INFO:scrape: 2023-10-22T01:42:10.258 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-10-22T01:42:10.259 INFO:scrape:2 jobs: ['7433026', '7433040'] 2023-10-22T01:42:10.259 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-10-22T01:42:10.259 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-10-22T01:42:10.260 INFO:scrape: 2023-10-22T01:42:10.262 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [7433026] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:08:05 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433026/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433026/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7323dc69af624fad8d611b51c2abaea7 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7433040] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:47 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433040/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433040/ sentry: https://sentry.ceph.com/organizations/ceph/?query=912ee60a29c34ff0b92ddd1546f52a84 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7433044] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:56:25 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433044/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433044/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b71b4b7b140479d90ecfdccefbed17c Command failed on smithi008 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb33f480-7059-11ee-8db6-212e2dc638e7 -e sha1=33809aa16833daf9cbc6787158e533522f8e6233 -- bash -c 'ceph orch ps'" Pass ================================================================= [7433011] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:47 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433011/ [7433013] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:21:09 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433013/ [7433015] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:41:55 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433015/ [7433017] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:55:54 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433017/ [7433019] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:24 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433019/ [7433021] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:37 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433021/ [7433023] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:03:56 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433023/ [7433025] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:23 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433025/ [7433027] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:08:49 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433027/ [7433028] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:26:14 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433028/ [7433029] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:12:36 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433029/ [7433030] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:09 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433030/ [7433031] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:51 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433031/ [7433032] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:05:03 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433032/ [7433033] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:24:25 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433033/ [7433034] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:57:28 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433034/ [7433035] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:58 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433035/ [7433036] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:06:14 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433036/ [7433037] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:20 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433037/ [7433038] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:28:38 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433038/ [7433039] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:48 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433039/ [7433041] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:44:50 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433041/ [7433042] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:24:42 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433042/ [7433043] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:30 info: http://pulpito.ceph.com/teuthology-2023-10-20_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7433043/