2022-03-28T09:52:56.525 INFO:root:teuthology version: 1.1.0-70390753 2022-03-28T09:52:56.526 INFO:teuthology.results:subset: '111/120000' 2022-03-28T09:52:56.526 INFO:teuthology.results:seed: '9080' 2022-03-28T10:57:51.025 INFO:root:teuthology version: 1.1.0-70390753 2022-03-28T10:57:51.062 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-28T14:11:04.894 INFO:teuthology.results:Tests finished! gathering results... 2022-03-28T14:11:07.435 INFO:scrape:Found 27 jobs 2022-03-28T14:11:10.030 WARNING:scrape:Not-understood crash indication 2022-03-28T11:18:10.438 INFO:tasks.rgw.c2.client.0.smithi189.stderr:daemon-helper: command crashed with signal 11 2022-03-28T14:11:10.031 WARNING:scrape:Not-understood crash indication 2022-03-28T14:11:10.049 WARNING:scrape:Not-understood crash indication 2022-03-28T11:18:10.438 INFO:tasks.rgw.c2.client.0.smithi189.stderr:daemon-helper: command crashed with signal 11 2022-03-28T14:11:10.050 WARNING:scrape:Not-understood crash indication 2022-03-28T14:11:10.071 WARNING:scrape:Not-understood crash indication 2022-03-28T11:18:10.438 INFO:tasks.rgw.c2.client.0.smithi189.stderr:daemon-helper: command crashed with signal 11 2022-03-28T14:11:10.072 WARNING:scrape:Not-understood crash indication 2022-03-28T14:11:10.090 WARNING:scrape:Not-understood crash indication 2022-03-28T11:18:10.438 INFO:tasks.rgw.c2.client.0.smithi189.stderr:daemon-helper: command crashed with signal 11 2022-03-28T14:11:10.091 WARNING:scrape:Not-understood crash indication 2022-03-28T14:11:10.314 WARNING:scrape:Not-understood crash indication 2022-03-28T10:41:50.304 INFO:tasks.rgw.c2.client.0.smithi101.stderr:daemon-helper: command crashed with signal 11 2022-03-28T14:11:10.315 WARNING:scrape:Not-understood crash indication 2022-03-28T14:11:10.333 WARNING:scrape:Not-understood crash indication 2022-03-28T10:41:50.304 INFO:tasks.rgw.c2.client.0.smithi101.stderr:daemon-helper: command crashed with signal 11 2022-03-28T14:11:10.334 WARNING:scrape:Not-understood crash indication 2022-03-28T14:11:10.335 INFO:scrape:Found 3 distinct failure reasons 2022-03-28T14:11:10.335 INFO:scrape:Failure: No module named 'tasks.rados' 2022-03-28T14:11:10.336 INFO:scrape:4 jobs: ['6764791', '6764795', '6764787', '6764797'] 2022-03-28T14:11:10.336 INFO:scrape:suites intersection: ['rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2022-03-28T14:11:10.337 INFO:scrape:suites union: ['0-roles', '1-ceph-install/octopus', '1-start', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8'] 2022-03-28T14:11:10.337 INFO:scrape: 2022-03-28T14:11:10.338 INFO:scrape:Failure: rgw multisite test failures 2022-03-28T14:11:10.338 INFO:scrape:2 jobs: ['6764793', '6764779'] 2022-03-28T14:11:10.339 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-03-28T14:11:10.339 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-03-28T14:11:10.340 INFO:scrape: 2022-03-28T14:11:10.340 INFO:scrape:Failure: Command failed on smithi149 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2022-03-28T14:11:10.340 INFO:scrape:2 jobs: ['6764781', '6764773'] 2022-03-28T14:11:10.341 INFO:scrape:suites intersection: ['1-ceph-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2022-03-28T14:11:10.341 INFO:scrape:suites union: ['1-ceph-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2022-03-28T14:11:10.342 INFO:scrape: 2022-03-28T14:11:10.344 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 19 Fail ================================================================= [6764773] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:42:04 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764773/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87d38d4e1aee403482c6ac710c2bf965 Command failed on smithi174 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6764779] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:06:46 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764779/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16cf2831b5304dbfa38490ac6ea6a972 rgw multisite test failures [6764781] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:31:04 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764781/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=840b40298bb3410e96fef2d05fac0497 Command failed on smithi149 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6764787] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:49:19 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28f3562b18fb40738ba93be8f3ea6a31 No module named 'tasks.rados' [6764791] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:38:50 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764791/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6ffc659bf114150b80321e7784c9a82 No module named 'tasks.rados' [6764793] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:07:20 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764793/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa530576095b40c5937f66aca15188e4 rgw multisite test failures [6764795] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:04:23 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764795/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764795/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8acd6d725ec495c82352786fcff182f No module named 'tasks.workunit' [6764797] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:33:59 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764797/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc74d8cfc0a14279ad8a7146faa07d3e No module named 'tasks.rados' Pass ================================================================= [6764771] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:09 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764771/ [6764772] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:54:11 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764772/ [6764774] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:03:42 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764774/ [6764775] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:47 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764775/ [6764776] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:18 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764776/ [6764777] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:24:04 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764777/ [6764778] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:03:01 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764778/ [6764780] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:45:13 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764780/ [6764782] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:14:36 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764782/ [6764783] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:03:35 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764783/ [6764784] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:58 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764784/ [6764785] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:23:42 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764785/ [6764786] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:18:53 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764786/ [6764788] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:15 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764788/ [6764789] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:58:26 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764789/ [6764790] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:40 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764790/ [6764792] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:16 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764792/ [6764794] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:38:49 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764794/ [6764796] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:02:49 info: http://pulpito.ceph.com/yuriw-2022-03-27_22:48:48-upgrade:octopus-x-quincy-distro-default-smithi/6764796/