2023-09-15T14:23:27.830 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-15T14:23:27.831 INFO:teuthology.results:subset: '111/120000' 2023-09-15T14:23:27.831 INFO:teuthology.results:seed: '323' 2023-09-15T14:24:22.084 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-15T14:24:22.088 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-16T02:25:13.150 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-09-16T02:25:16.300 INFO:scrape:Found 27 jobs 2023-09-16T02:25:16.448 INFO:scrape:Found 2 distinct failure reasons 2023-09-16T02:25:16.449 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-09-16T02:25:16.449 INFO:scrape:2 jobs: ['7397097', '7397118'] 2023-09-16T02:25:16.450 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-09-16T02:25:16.450 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-09-16T02:25:16.451 INFO:scrape: 2023-09-16T02:25:16.451 INFO:scrape:Dead: b'2023-09-15T15:35:56.607 INFO:tasks.qemu.client.0.smithi158.stdout:[ 1320.222726] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.' 2023-09-16T02:25:16.451 INFO:scrape:1 jobs: ['7397098'] 2023-09-16T02:25:16.452 INFO:scrape:suites: ['1-octopus-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-hybrid', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-quincy', '7-final-workload', 'mon_election/classic', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack'] 2023-09-16T02:25:16.452 INFO:scrape: 2023-09-16T02:25:16.454 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 2 dead: 0 running: 1 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [7397097] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:10:26 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397097/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5888d7d53e5485fb1d733f218951517 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7397118] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:08:21 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397118/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397118/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f12b7fb37922418c9cd94bd0d6a92332 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus Running ================================================================= [7397098] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397098/ Pass ================================================================= [7397089] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:53 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397089/ [7397090] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:56:53 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397090/ [7397091] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:35:10 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397091/ [7397092] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:59:07 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397092/ [7397093] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:11:04 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397093/ [7397094] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:18 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397094/ [7397095] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:12:19 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397095/ [7397096] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:38 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397096/ [7397099] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:19:36 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397099/ [7397100] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:15:45 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397100/ [7397101] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:49 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397101/ [7397102] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:30 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397102/ [7397103] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:11:48 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397103/ [7397105] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:19:57 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397105/ [7397107] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:44:45 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397107/ [7397109] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:48 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397109/ [7397111] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:59:34 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397111/ [7397113] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:46 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397113/ [7397115] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:18:53 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397115/ [7397116] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:35 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397116/ [7397120] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:45:15 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397120/ [7397122] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:21:44 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397122/ [7397125] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:40 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397125/ [7397126] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:59:32 info: http://pulpito.ceph.com/teuthology-2023-09-15_14:22:10-upgrade:octopus-x-quincy-distro-default-smithi/7397126/