2023-04-08T00:13:16.093 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-08T00:13:16.094 INFO:teuthology.results:subset: '111/120000' 2023-04-08T00:13:16.094 INFO:teuthology.results:seed: '327' 2023-04-08T01:47:12.013 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-08T01:47:12.042 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-08T13:43:00.985 INFO:teuthology.results:Tests finished! gathering results... 2023-04-08T13:43:03.823 INFO:scrape:Found 27 jobs 2023-04-08T13:43:07.707 INFO:scrape:Found 4 distinct failure reasons 2023-04-08T13:43:07.708 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-04-08T13:43:07.708 INFO:scrape:2 jobs: ['7235108', '7235132'] 2023-04-08T13:43:07.709 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-04-08T13:43:07.709 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-04-08T13:43:07.709 INFO:scrape: 2023-04-08T13:43:07.709 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-04-08T13:43:07.710 INFO:scrape:1 jobs: ['7235105'] 2023-04-08T13:43:07.710 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8'] 2023-04-08T13:43:07.710 INFO:scrape: 2023-04-08T13:43:07.711 INFO:scrape:Failure: Command failed on smithi145 with status 1: 'sudo yum install -y kernel' 2023-04-08T13:43:07.711 INFO:scrape:1 jobs: ['7235135'] 2023-04-08T13:43:07.711 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'workload/{ec-rados-default'] 2023-04-08T13:43:07.711 INFO:scrape: 2023-04-08T13:43:07.712 INFO:scrape:Dead: b'2023-04-08T12:51:32.474 DEBUG:teuthology.exit:Finished running handlers' 2023-04-08T13:43:07.712 INFO:scrape:2 jobs: ['7235120', '7235128'] 2023-04-08T13:43:07.712 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2023-04-08T13:43:07.713 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun'] 2023-04-08T13:43:07.713 INFO:scrape: 2023-04-08T13:43:07.715 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 4 dead: 2 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [7235105] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 02:13:21 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235105/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b23fcacefaba4605954ebf154ecea0a3 timeout expired in wait_until_healthy [7235108] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:27 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235108/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235108/ sentry: https://sentry.ceph.com/organizations/ceph/?query=226d35e69d744a93801ac550b03be33d Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7235132] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:23 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235132/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235132/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f41fa6cde7ee48e09dbd89184683563d Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7235135] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235135/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235135/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca26abc9975b4437bffe6add946d8074 Command failed on smithi145 with status 1: 'sudo yum install -y kernel' Dead ================================================================= [7235120] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235120/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235120/ hit max job timeout [7235128] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235128/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235128/ hit max job timeout Pass ================================================================= [7235099] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:15 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235099/ [7235100] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:51:44 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235100/ [7235101] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:33:27 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235101/ [7235102] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:49:41 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235102/ [7235103] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:29 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235103/ [7235104] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:30 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235104/ [7235106] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 02:03:26 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235106/ [7235110] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:18:51 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235110/ [7235112] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:02:37 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235112/ [7235114] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:59:36 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235114/ [7235116] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 02:03:54 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235116/ [7235118] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:26 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235118/ [7235122] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:14:04 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235122/ [7235124] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:41:14 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235124/ [7235126] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:22 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235126/ [7235129] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:02 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235129/ [7235130] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:03:58 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235130/ [7235131] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 02:04:38 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235131/ [7235133] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:41:27 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235133/ [7235134] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 03:55:05 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235134/ [7235136] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:54:52 info: http://pulpito.ceph.com/teuthology-2023-04-07_14:22:12-upgrade:octopus-x-quincy-distro-default-smithi/7235136/