2023-04-23T06:50:35.455 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-23T06:50:35.456 INFO:teuthology.results:subset: '111/120000' 2023-04-23T06:50:35.456 INFO:teuthology.results:seed: '6593' 2023-04-23T10:09:51.762 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-23T10:09:51.820 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-23T20:32:40.382 INFO:teuthology.results:Tests finished! gathering results... 2023-04-23T20:32:43.500 INFO:scrape:Found 27 jobs 2023-04-23T20:32:46.385 INFO:scrape:Found 3 distinct failure reasons 2023-04-23T20:32:46.386 INFO:scrape:Dead: b'2023-04-23T20:29:35.204 DEBUG:teuthology.exit:Finished running handlers' 2023-04-23T20:32:46.386 INFO:scrape:1 jobs: ['7247711'] 2023-04-23T20:32:46.387 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'workload/{ec-rados-default'] 2023-04-23T20:32:46.387 INFO:scrape: 2023-04-23T20:32:46.387 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-04-23T20:32:46.387 INFO:scrape:2 jobs: ['7247725', '7247707'] 2023-04-23T20:32:46.388 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-04-23T20:32:46.388 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-04-23T20:32:46.388 INFO:scrape: 2023-04-23T20:32:46.389 INFO:scrape:Failure: Found coredumps on ubuntu@smithi073.front.sepia.ceph.com 2023-04-23T20:32:46.389 INFO:scrape:1 jobs: ['7247717'] 2023-04-23T20:32:46.389 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun'] 2023-04-23T20:32:46.390 INFO:scrape: 2023-04-23T20:32:46.392 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 1 running: 0 waiting: 0 queued: 0 passed: 23 Fail ================================================================= [7247707] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:34 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247707/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247707/ sentry: https://sentry.ceph.com/organizations/ceph/?query=344e91028e5e4107af39478918c29d45 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7247717] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:17:18 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247717/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247717/ Found coredumps on ubuntu@smithi073.front.sepia.ceph.com [7247725] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:36 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247725/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f6f54762b0847079668764f8529bbd4 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Dead ================================================================= [7247711] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247711/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247711/ hit max job timeout Pass ================================================================= [7247699] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:18:01 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247699/ [7247700] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:23:18 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247700/ [7247701] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:39:48 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247701/ [7247702] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:12:09 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247702/ [7247703] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:11 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247703/ [7247704] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:42 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247704/ [7247705] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:12:01 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247705/ [7247706] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 02:27:29 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247706/ [7247708] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:44:59 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247708/ [7247709] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:02:04 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247709/ [7247710] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:01:26 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247710/ [7247712] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:57 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247712/ [7247713] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 03:46:27 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247713/ [7247714] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:20:08 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247714/ [7247715] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:41:25 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247715/ [7247716] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:11:20 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247716/ [7247719] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:35 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247719/ [7247721] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:01:39 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247721/ [7247723] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 02:24:13 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247723/ [7247727] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:43:31 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247727/ [7247729] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:10:19 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247729/ [7247731] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 02:27:16 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247731/ [7247733] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:05:33 info: http://pulpito.ceph.com/teuthology-2023-04-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7247733/