2022-06-17T14:22:23.224 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-17T14:22:23.227 INFO:teuthology.results:subset: '4/10' 2022-06-17T14:22:23.227 INFO:teuthology.results:seed: '8144' 2022-06-17T14:39:01.522 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-17T14:39:01.562 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-17T19:46:29.199 INFO:teuthology.results:Tests finished! gathering results... 2022-06-17T19:46:32.582 INFO:scrape:Found 32 jobs 2022-06-17T19:46:32.810 WARNING:scrape:Not-understood crash indication 2022-06-17T15:10:09.750 INFO:tasks.rgw.c2.client.0.smithi089.stderr:daemon-helper: command crashed with signal 11 2022-06-17T19:46:32.811 WARNING:scrape:Not-understood crash indication 2022-06-17T19:46:32.831 WARNING:scrape:Not-understood crash indication 2022-06-17T15:10:09.750 INFO:tasks.rgw.c2.client.0.smithi089.stderr:daemon-helper: command crashed with signal 11 2022-06-17T19:46:32.832 WARNING:scrape:Not-understood crash indication 2022-06-17T19:46:32.852 WARNING:scrape:Not-understood crash indication 2022-06-17T15:10:09.750 INFO:tasks.rgw.c2.client.0.smithi089.stderr:daemon-helper: command crashed with signal 11 2022-06-17T19:46:32.853 WARNING:scrape:Not-understood crash indication 2022-06-17T19:46:33.059 WARNING:scrape:Not-understood crash indication 2022-06-17T14:58:33.076 INFO:tasks.rgw.c2.client.0.smithi169.stderr:daemon-helper: command crashed with signal 11 2022-06-17T19:46:33.060 WARNING:scrape:Not-understood crash indication 2022-06-17T19:46:33.061 INFO:scrape:Found 1 distinct failure reasons 2022-06-17T19:46:33.062 INFO:scrape:Failure: rgw multisite test failures 2022-06-17T19:46:33.062 INFO:scrape:2 jobs: ['6884426', '6884410'] 2022-06-17T19:46:33.063 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-06-17T19:46:33.063 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-06-17T19:46:33.064 INFO:scrape: 2022-06-17T19:46:33.066 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [6884410] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 02:17:35 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884410/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13d210e073d44e598cdabc6e0267a9b3 rgw multisite test failures [6884426] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 02:19:30 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884426/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f23971e1a5d449288c543d3c040f38c8 rgw multisite test failures Pass ================================================================= [6884409] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:08 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884409/ [6884411] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:29:52 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884411/ [6884412] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:01:48 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884412/ [6884413] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 05:07:58 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884413/ [6884414] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:16:28 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884414/ [6884415] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:19 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884415/ [6884416] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:00:31 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884416/ [6884417] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:31 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884417/ [6884418] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:55:38 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884418/ [6884419] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 01:47:27 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884419/ [6884420] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:37:02 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884420/ [6884421] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:06 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884421/ [6884422] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:13:03 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884422/ [6884423] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:13:36 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884423/ [6884424] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:26:50 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884424/ [6884425] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:25 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884425/ [6884427] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:26:55 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884427/ [6884428] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:22:18 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884428/ [6884429] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:57:46 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884429/ [6884430] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:57:05 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884430/ [6884431] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:38 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884431/ [6884432] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:16:16 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884432/ [6884433] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:19:44 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884433/ [6884434] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:07:43 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884434/ [6884435] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:34:29 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884435/ [6884436] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:40:05 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884436/ [6884437] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:32 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884437/ [6884438] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:08:52 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884438/ [6884439] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:12:13 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884439/ [6884440] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:11:53 info: http://pulpito.ceph.com/teuthology-2022-06-17_14:22:04-upgrade:octopus-x-quincy-distro-default-smithi/6884440/