2024-04-15T02:08:16.175 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-15T02:08:16.175 INFO:teuthology.results:subset: '23299/120000' 2024-04-15T02:08:16.175 INFO:teuthology.results:seed: '7378' 2024-04-15T02:26:23.682 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-15T02:26:23.703 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-15T06:59:45.019 INFO:teuthology.results:Tests finished! gathering results... 2024-04-15T06:59:47.517 INFO:scrape:Found 27 jobs 2024-04-15T06:59:47.518 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656985/teuthology.log 2024-04-15T06:59:48.126 INFO:scrape:Found 3 distinct failure reasons 2024-04-15T06:59:48.126 INFO:scrape:Dead: None 2024-04-15T06:59:48.127 INFO:scrape:1 jobs: ['7656985'] 2024-04-15T06:59:48.127 INFO:scrape: 2024-04-15T06:59:48.127 INFO:scrape:Failure: Command failed on smithi106 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9451350-facf-11ee-bc8e-c7b262605968 -e sha1=4f113f5455944ac06c5d93df586ceb9be0f9dfff -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2024-04-15T06:59:48.127 INFO:scrape:1 jobs: ['7656970'] 2024-04-15T06:59:48.127 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'workload/{ec-rados-default'] 2024-04-15T06:59:48.127 INFO:scrape: 2024-04-15T06:59:48.127 INFO:scrape:Failure: An attempt to upgrade from a higher version to a lower one will always fail. Hint: check tags in the target git branch. 2024-04-15T06:59:48.127 INFO:scrape:2 jobs: ['7656971', '7656984'] 2024-04-15T06:59:48.127 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2024-04-15T06:59:48.127 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2024-04-15T06:59:48.128 INFO:scrape: 2024-04-15T06:59:48.130 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 1 running: 0 waiting: 0 queued: 0 passed: 23 Fail ================================================================= [7656970] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:56:39 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656970/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656970/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd7481c1bd7343d58817eb21a063f0f4 Command failed on smithi106 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9451350-facf-11ee-bc8e-c7b262605968 -e sha1=4f113f5455944ac06c5d93df586ceb9be0f9dfff -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7656971] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:10:06 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656971/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656971/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8d0d82247824eb4a3dff24f26b4f625 An attempt to upgrade from a higher version to a lower one will always fail. Hint: check tags in the target git branch. [7656984] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:09:59 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656984/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656984/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb757b4131574dd3a931d0df83a3c2af An attempt to upgrade from a higher version to a lower one will always fail. Hint: check tags in the target git branch. Dead ================================================================= [7656985] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656985/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656985/ Error reimaging machines: Failed to power on smithi055 Pass ================================================================= [7656961] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656961/ [7656962] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 03:29:10 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656962/ [7656963] upgrade:octopus-x/parallel/{0-distro/rhel_8.6_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:09 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656963/ [7656964] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 01:37:42 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656964/ [7656965] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:11:35 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656965/ [7656966] upgrade:octopus-x/parallel/{0-distro/rhel_8.6_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:20 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656966/ [7656967] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:57:01 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656967/ [7656968] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:03 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656968/ [7656969] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:10:38 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656969/ [7656972] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:03:56 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656972/ [7656973] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 03:48:46 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656973/ [7656974] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:47 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656974/ [7656975] upgrade:octopus-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:13:22 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656975/ [7656976] upgrade:octopus-x/parallel/{0-distro/rhel_8.6_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:47 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656976/ [7656977] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:08:20 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656977/ [7656978] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 03:28:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656978/ [7656979] upgrade:octopus-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:01:18 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656979/ [7656980] upgrade:octopus-x/parallel/{0-distro/rhel_8.6_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656980/ [7656981] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:12 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656981/ [7656982] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:23:06 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656982/ [7656983] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:58 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656983/ [7656986] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:23:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656986/ [7656987] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:08:31 info: https://pulpito.ceph.com/teuthology-2024-04-15_00:24:02-upgrade:octopus-x-quincy-distro-default-smithi/7656987/