2022-04-14T22:35:05.651 INFO:root:teuthology version: 1.1.0-0e6dbbf1 2022-04-14T22:35:05.652 INFO:teuthology.results:subset: '111/120000' 2022-04-14T22:35:05.652 INFO:teuthology.results:seed: '2027' 2022-04-14T22:57:41.500 INFO:root:teuthology version: 1.1.0-0e6dbbf1 2022-04-14T22:57:41.658 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-15T03:33:02.118 INFO:teuthology.results:Tests finished! gathering results... 2022-04-15T03:33:05.158 INFO:scrape:Found 27 jobs 2022-04-15T03:33:05.342 WARNING:scrape:Not-understood crash indication 2022-04-14T23:11:16.402 INFO:tasks.rgw.c2.client.0.smithi134.stderr:daemon-helper: command crashed with signal 11 2022-04-15T03:33:05.343 WARNING:scrape:Not-understood crash indication 2022-04-15T03:33:05.364 WARNING:scrape:Not-understood crash indication 2022-04-14T23:11:16.402 INFO:tasks.rgw.c2.client.0.smithi134.stderr:daemon-helper: command crashed with signal 11 2022-04-15T03:33:05.364 WARNING:scrape:Not-understood crash indication 2022-04-15T03:33:05.383 WARNING:scrape:Not-understood crash indication 2022-04-14T23:11:16.402 INFO:tasks.rgw.c2.client.0.smithi134.stderr:daemon-helper: command crashed with signal 11 2022-04-15T03:33:05.383 WARNING:scrape:Not-understood crash indication 2022-04-15T03:33:05.401 WARNING:scrape:Not-understood crash indication 2022-04-14T23:11:16.402 INFO:tasks.rgw.c2.client.0.smithi134.stderr:daemon-helper: command crashed with signal 11 2022-04-15T03:33:05.401 WARNING:scrape:Not-understood crash indication 2022-04-15T03:33:05.545 WARNING:scrape:Not-understood crash indication 2022-04-14T23:22:41.710 INFO:tasks.rgw.c2.client.0.smithi040.stderr:daemon-helper: command crashed with signal 11 2022-04-15T03:33:05.545 WARNING:scrape:Not-understood crash indication 2022-04-15T03:33:05.568 WARNING:scrape:Not-understood crash indication 2022-04-14T23:22:41.710 INFO:tasks.rgw.c2.client.0.smithi040.stderr:daemon-helper: command crashed with signal 11 2022-04-15T03:33:05.569 WARNING:scrape:Not-understood crash indication 2022-04-15T03:33:05.569 INFO:scrape:Found 2 distinct failure reasons 2022-04-15T03:33:05.569 INFO:scrape:Failure: Command failed on smithi196 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 2022-04-15T03:33:05.569 INFO:scrape:1 jobs: ['6791608'] 2022-04-15T03:33:05.570 INFO:scrape:suites: ['1-octopus-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-hybrid', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-quincy', '7-final-workload', 'mon_election/classic', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack'] 2022-04-15T03:33:05.570 INFO:scrape: 2022-04-15T03:33:05.570 INFO:scrape:Failure: rgw multisite test failures 2022-04-15T03:33:05.570 INFO:scrape:2 jobs: ['6791605', '6791634'] 2022-04-15T03:33:05.570 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-04-15T03:33:05.570 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-04-15T03:33:05.570 INFO:scrape: 2022-04-15T03:33:05.572 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [6791605] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:06:14 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791605/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791605/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07da51ff96164dc597d3e20879aca9ae rgw multisite test failures [6791608] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:19:57 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791608/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791608/ Command failed on smithi196 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' [6791634] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:08:35 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791634/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791634/ sentry: https://sentry.ceph.com/organizations/ceph/?query=084a89e62e5745b39e114623f65c6d1d rgw multisite test failures Pass ================================================================= [6791590] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:14:45 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791590/ [6791592] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:25:36 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791592/ [6791594] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:41:03 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791594/ [6791596] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:04:03 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791596/ [6791598] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:24 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791598/ [6791600] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:08 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791600/ [6791602] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:11:38 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791602/ [6791603] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:45 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791603/ [6791610] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:28:48 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791610/ [6791612] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:02:18 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791612/ [6791613] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:28 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791613/ [6791615] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:14:49 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791615/ [6791617] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:17:47 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791617/ [6791620] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:19:22 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791620/ [6791622] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:45:09 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791622/ [6791624] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:53 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791624/ [6791626] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:08:48 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791626/ [6791628] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:52 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791628/ [6791630] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:12:07 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791630/ [6791632] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:45 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791632/ [6791636] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:38:56 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791636/ [6791638] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:29:15 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791638/ [6791640] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:30 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791640/ [6791642] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:10:23 info: http://pulpito.ceph.com/yuriw-2022-04-14_22:31:02-upgrade:octopus-x-quincy-distro-default-smithi/6791642/