2021-11-26T14:23:36.655 INFO:root:teuthology version: 1.1.0-a0bc3188 2021-11-26T14:23:36.659 INFO:teuthology.results:subset: '6/10' 2021-11-26T14:23:36.659 INFO:teuthology.results:seed: '3757' 2021-11-26T14:56:58.411 INFO:root:teuthology version: 1.1.0-a0bc3188 2021-11-26T14:56:58.435 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-11-26T20:40:25.364 INFO:teuthology.results:Tests finished! gathering results... 2021-11-26T20:40:29.854 INFO:scrape:Found 40 jobs 2021-11-26T20:40:30.040 WARNING:scrape:Not-understood crash indication 2021-11-26T15:17:42.488 INFO:tasks.rgw.c2.client.0.smithi140.stderr:daemon-helper: command crashed with signal 11 2021-11-26T20:40:30.041 WARNING:scrape:Not-understood crash indication 2021-11-26T20:40:30.075 WARNING:scrape:Not-understood crash indication 2021-11-26T15:17:42.488 INFO:tasks.rgw.c2.client.0.smithi140.stderr:daemon-helper: command crashed with signal 11 2021-11-26T20:40:30.076 WARNING:scrape:Not-understood crash indication 2021-11-26T20:40:30.102 WARNING:scrape:Not-understood crash indication 2021-11-26T15:17:42.488 INFO:tasks.rgw.c2.client.0.smithi140.stderr:daemon-helper: command crashed with signal 11 2021-11-26T20:40:30.103 WARNING:scrape:Not-understood crash indication 2021-11-26T20:40:30.243 WARNING:scrape:Not-understood crash indication 2021-11-26T15:03:29.514 INFO:tasks.rgw.c2.client.0.smithi158.stderr:daemon-helper: command crashed with signal 11 2021-11-26T20:40:30.244 WARNING:scrape:Not-understood crash indication 2021-11-26T20:40:30.245 INFO:scrape:Found 1 distinct failure reasons 2021-11-26T20:40:30.245 INFO:scrape:Failure: rgw multisite test failures 2021-11-26T20:40:30.246 INFO:scrape:2 jobs: ['6528334', '6528313'] 2021-11-26T20:40:30.246 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2021-11-26T20:40:30.246 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2021-11-26T20:40:30.247 INFO:scrape: 2021-11-26T20:40:30.250 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [6528313] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:04:05 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528313/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d87055ff511461ca309b48353ee4761 rgw multisite test failures [6528334] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:04:21 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528334/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528334/ sentry: https://sentry.ceph.com/organizations/ceph/?query=60c7257e5ec34f41843b10c0f1146c0f rgw multisite test failures Pass ================================================================= [6528302] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:58:52 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528302/ [6528304] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:19:00 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528304/ [6528306] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:01:30 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528306/ [6528308] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:56 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528308/ [6528310] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:56:55 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528310/ [6528312] upgrade:octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:54 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528312/ [6528314] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:50:03 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528314/ [6528315] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} time: 05:32:25 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528315/ [6528316] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:09:52 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528316/ [6528317] upgrade:octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:31 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528317/ [6528318] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:24:47 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528318/ [6528319] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:30:36 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528319/ [6528320] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:19 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528320/ [6528321] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:13:50 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528321/ [6528322] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:11 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528322/ [6528323] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:00:43 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528323/ [6528324] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:46:15 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528324/ [6528325] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:30:24 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528325/ [6528326] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:46 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528326/ [6528327] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:53:50 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528327/ [6528328] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:48 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528328/ [6528329] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:07:31 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528329/ [6528330] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:16 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528330/ [6528331] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:58:47 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528331/ [6528332] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:18:51 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528332/ [6528333] upgrade:octopus-x/parallel/{0-distro/centos_8.2_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:46 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528333/ [6528335] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 00:48:56 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528335/ [6528336] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_20.04} time: 05:45:22 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528336/ [6528337] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:07:01 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528337/ [6528338] upgrade:octopus-x/parallel/{0-distro/centos_8.3_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:19 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528338/ [6528339] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:17:38 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528339/ [6528340] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:55 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528340/ [6528341] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:01:01 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528341/ [6528342] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:37 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528342/ [6528343] upgrade:octopus-x/stress-split/{0-distro/centos_8.3_container_tools_3.0 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:32:54 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528343/ [6528344] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 00:48:33 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528344/ [6528345] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:30:34 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528345/ [6528346] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:32 info: http://pulpito.ceph.com/teuthology-2021-11-26_14:22:03-upgrade:octopus-x-master-distro-default-smithi/6528346/