2022-03-18T14:22:44.408 INFO:root:teuthology version: 1.1.0-aa078dbe 2022-03-18T14:22:44.409 INFO:teuthology.results:subset: '3/10' 2022-03-18T14:22:44.410 INFO:teuthology.results:seed: '8233' 2022-03-18T17:00:58.426 INFO:root:teuthology version: 1.1.0-aa078dbe 2022-03-18T17:00:58.443 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-18T21:17:17.067 INFO:teuthology.results:Tests finished! gathering results... 2022-03-18T21:17:20.275 INFO:scrape:Found 32 jobs 2022-03-18T21:17:20.476 WARNING:scrape:Not-understood crash indication 2022-03-18T15:33:03.462 INFO:tasks.rgw.c2.client.0.smithi173.stderr:daemon-helper: command crashed with signal 11 2022-03-18T21:17:20.477 WARNING:scrape:Not-understood crash indication 2022-03-18T21:17:20.500 WARNING:scrape:Not-understood crash indication 2022-03-18T15:33:03.462 INFO:tasks.rgw.c2.client.0.smithi173.stderr:daemon-helper: command crashed with signal 11 2022-03-18T21:17:20.501 WARNING:scrape:Not-understood crash indication 2022-03-18T21:17:20.522 WARNING:scrape:Not-understood crash indication 2022-03-18T15:33:03.462 INFO:tasks.rgw.c2.client.0.smithi173.stderr:daemon-helper: command crashed with signal 11 2022-03-18T21:17:20.523 WARNING:scrape:Not-understood crash indication 2022-03-18T21:17:20.558 WARNING:scrape:Not-understood crash indication 2022-03-18T15:33:03.462 INFO:tasks.rgw.c2.client.0.smithi173.stderr:daemon-helper: command crashed with signal 11 2022-03-18T21:17:20.559 WARNING:scrape:Not-understood crash indication 2022-03-18T21:17:20.691 WARNING:scrape:Not-understood crash indication 2022-03-18T15:00:09.195 INFO:tasks.rgw.c2.client.0.smithi145.stderr:daemon-helper: command crashed with signal 11 2022-03-18T21:17:20.692 WARNING:scrape:Not-understood crash indication 2022-03-18T21:17:20.694 INFO:scrape:Found 3 distinct failure reasons 2022-03-18T21:17:20.695 INFO:scrape:Failure: Command failed on smithi161 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2022-03-18T21:17:20.696 INFO:scrape:4 jobs: ['6745203', '6745180', '6745187', '6745196'] 2022-03-18T21:17:20.696 INFO:scrape:suites intersection: ['1-ceph-install/octopus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2022-03-18T21:17:20.697 INFO:scrape:suites union: ['1-ceph-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2022-03-18T21:17:20.697 INFO:scrape: 2022-03-18T21:17:20.698 INFO:scrape:Failure: rgw multisite test failures 2022-03-18T21:17:20.698 INFO:scrape:1 jobs: ['6745193'] 2022-03-18T21:17:20.699 INFO:scrape:suites: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-03-18T21:17:20.699 INFO:scrape: 2022-03-18T21:17:20.700 INFO:scrape:Dead: b'2022-03-18T21:02:35.485 DEBUG:teuthology.exit:Finished running handlers' 2022-03-18T21:17:20.700 INFO:scrape:1 jobs: ['6745177'] 2022-03-18T21:17:20.700 INFO:scrape:suites: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-03-18T21:17:20.701 INFO:scrape: 2022-03-18T21:17:20.704 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 5 dead: 1 running: 0 waiting: 0 queued: 0 passed: 26 Fail ================================================================= [6745180] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:29:44 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce5ea573c56b433187637f668ef22f00 Command failed on smithi159 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6745187] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 02:03:36 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745187/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=683d849aee8349f685fcff72f7f2ad1b Command failed on smithi188 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6745193] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:08:00 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745193/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bbecd50ff41c44bc87225aa536aeb240 rgw multisite test failures [6745196] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:37:46 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745196/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70bd9668b99540b182663809d8ceb123 Command failed on smithi186 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6745203] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:55:46 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745203/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=237ff3634d684d2885784702d8572397 Command failed on smithi161 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' Dead ================================================================= [6745177] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745177/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745177/ hit max job timeout Pass ================================================================= [6745176] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:24 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745176/ [6745178] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:19:29 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745178/ [6745179] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:34:56 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745179/ [6745181] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:08:16 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745181/ [6745182] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:06 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745182/ [6745183] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:12:40 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745183/ [6745184] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:59 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745184/ [6745185] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:00:54 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745185/ [6745186] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:13:58 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745186/ [6745188] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:15:39 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745188/ [6745189] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:09:52 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745189/ [6745190] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:02 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745190/ [6745191] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:49:20 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745191/ [6745192] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:36 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745192/ [6745194] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:22:58 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745194/ [6745195] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:39:28 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745195/ [6745197] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:05:02 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745197/ [6745198] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:54 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745198/ [6745199] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:18:50 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745199/ [6745200] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:18:17 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745200/ [6745201] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:01:17 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745201/ [6745202] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 04:03:16 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745202/ [6745204] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:01 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745204/ [6745205] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:08:40 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745205/ [6745206] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:49 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745206/ [6745207] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:10:19 info: http://pulpito.ceph.com/teuthology-2022-03-18_14:22:02-upgrade:octopus-x-quincy-distro-default-smithi/6745207/