2021-12-03T14:24:13.029 INFO:root:teuthology version: 1.1.0-844d1377 2021-12-03T14:24:13.031 INFO:teuthology.results:subset: '2/10' 2021-12-03T14:24:13.031 INFO:teuthology.results:seed: '3585' 2021-12-03T15:00:39.152 INFO:root:teuthology version: 1.1.0-844d1377 2021-12-03T15:00:39.181 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-12-03T21:17:08.424 INFO:teuthology.results:Tests finished! gathering results... 2021-12-03T21:17:12.594 INFO:scrape:Found 40 jobs 2021-12-03T21:17:12.767 WARNING:scrape:Not-understood crash indication 2021-12-03T15:01:57.608 INFO:tasks.rgw.c2.client.0.smithi183.stderr:daemon-helper: command crashed with signal 11 2021-12-03T21:17:12.768 WARNING:scrape:Not-understood crash indication 2021-12-03T21:17:12.795 WARNING:scrape:Not-understood crash indication 2021-12-03T15:01:57.608 INFO:tasks.rgw.c2.client.0.smithi183.stderr:daemon-helper: command crashed with signal 11 2021-12-03T21:17:12.796 WARNING:scrape:Not-understood crash indication 2021-12-03T21:17:12.821 WARNING:scrape:Not-understood crash indication 2021-12-03T15:01:57.608 INFO:tasks.rgw.c2.client.0.smithi183.stderr:daemon-helper: command crashed with signal 11 2021-12-03T21:17:12.823 WARNING:scrape:Not-understood crash indication 2021-12-03T21:17:12.958 WARNING:scrape:Not-understood crash indication 2021-12-03T15:11:38.030 INFO:tasks.rgw.c2.client.0.smithi085.stderr:daemon-helper: command crashed with signal 11 2021-12-03T21:17:12.959 WARNING:scrape:Not-understood crash indication 2021-12-03T21:17:12.960 INFO:scrape:Found 1 distinct failure reasons 2021-12-03T21:17:12.961 INFO:scrape:Failure: rgw multisite test failures 2021-12-03T21:17:12.961 INFO:scrape:2 jobs: ['6542513', '6542533'] 2021-12-03T21:17:12.962 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2021-12-03T21:17:12.963 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2021-12-03T21:17:12.964 INFO:scrape: 2021-12-03T21:17:12.969 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [6542513] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:03:48 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542513/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542513/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b81a90095f25499abbb56bd007d37002 rgw multisite test failures [6542533] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:04:05 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542533/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542533/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3596386e007b4ff685c0f0dc73c3358b rgw multisite test failures Pass ================================================================= [6542510] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:41 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542510/ [6542511] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:12:59 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542511/ [6542512] upgrade:octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:05 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542512/ [6542514] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:48:32 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542514/ [6542515] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 06:37:37 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542515/ [6542516] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:16:56 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542516/ [6542517] upgrade:octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:59 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542517/ [6542518] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:57:19 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542518/ [6542519] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:31 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542519/ [6542520] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:22:55 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542520/ [6542521] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:54:00 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542521/ [6542522] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 00:59:39 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542522/ [6542523] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:45:01 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542523/ [6542524] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:38:43 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542524/ [6542525] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 01:55:44 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542525/ [6542526] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:58 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542526/ [6542527] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:11:07 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542527/ [6542528] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:47 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542528/ [6542529] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:14:52 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542529/ [6542530] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:44 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542530/ [6542531] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:05:20 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542531/ [6542532] upgrade:octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:01 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542532/ [6542534] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:19:20 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542534/ [6542535] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:46:13 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542535/ [6542536] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 06:17:42 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542536/ [6542537] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:04:21 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542537/ [6542538] upgrade:octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:54 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542538/ [6542539] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 01:54:18 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542539/ [6542540] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 00:59:11 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542540/ [6542541] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:51:53 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542541/ [6542542] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 00:59:12 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542542/ [6542543] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:16:36 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542543/ [6542544] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:47:16 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542544/ [6542545] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:27:54 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542545/ [6542546] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:54 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542546/ [6542547] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:17:09 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542547/ [6542548] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:18:15 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542548/ [6542549] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:34:11 info: http://pulpito.ceph.com/teuthology-2021-12-03_14:22:02-upgrade:octopus-x-master-distro-default-smithi/6542549/