2022-05-20T22:44:07.709 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-20T22:44:07.711 INFO:teuthology.results:subset: '1/30' 2022-05-20T22:44:07.711 INFO:teuthology.results:seed: '3759' 2022-05-20T23:45:41.312 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-20T23:45:41.324 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-21T03:42:00.300 INFO:teuthology.results:Tests finished! gathering results... 2022-05-21T03:42:02.897 INFO:scrape:Found 28 jobs 2022-05-21T03:42:03.104 WARNING:scrape:Not-understood crash indication 2022-05-20T23:43:03.550 INFO:tasks.rgw.c2.client.0.smithi119.stderr:daemon-helper: command crashed with signal 11 2022-05-21T03:42:03.105 WARNING:scrape:Not-understood crash indication 2022-05-21T03:42:03.124 WARNING:scrape:Not-understood crash indication 2022-05-20T23:43:03.550 INFO:tasks.rgw.c2.client.0.smithi119.stderr:daemon-helper: command crashed with signal 11 2022-05-21T03:42:03.125 WARNING:scrape:Not-understood crash indication 2022-05-21T03:42:03.147 WARNING:scrape:Not-understood crash indication 2022-05-20T23:43:03.550 INFO:tasks.rgw.c2.client.0.smithi119.stderr:daemon-helper: command crashed with signal 11 2022-05-21T03:42:03.148 WARNING:scrape:Not-understood crash indication 2022-05-21T03:42:03.360 WARNING:scrape:Not-understood crash indication 2022-05-20T23:23:39.180 INFO:tasks.rgw.c2.client.0.smithi157.stderr:daemon-helper: command crashed with signal 11 2022-05-21T03:42:03.361 WARNING:scrape:Not-understood crash indication 2022-05-21T03:42:03.362 INFO:scrape:Found 1 distinct failure reasons 2022-05-21T03:42:03.362 INFO:scrape:Failure: rgw multisite test failures 2022-05-21T03:42:03.363 INFO:scrape:2 jobs: ['6843040', '6843026'] 2022-05-21T03:42:03.363 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-05-21T03:42:03.364 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-05-21T03:42:03.364 INFO:scrape: 2022-05-21T03:42:03.366 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 26 Fail ================================================================= [6843026] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 02:15:46 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843026/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843026/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b47f9a55e2343908db0aea087c20717 rgw multisite test failures [6843040] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 02:16:22 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843040/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843040/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cde5ed3235b34b20afdb3ec2de6f3b62 rgw multisite test failures Pass ================================================================= [6843025] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:03:53 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843025/ [6843027] upgrade:octopus-x/stress-split/{0-distro/rhel_8.5_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:03:42 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843027/ [6843028] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 03:26:32 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843028/ [6843029] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:30:55 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843029/ [6843030] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:32 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843030/ [6843031] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:07:22 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843031/ [6843032] upgrade:octopus-x/parallel/{0-distro/rhel_8.5_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:15 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843032/ [6843033] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 01:56:09 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843033/ [6843034] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:11:36 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843034/ [6843035] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:47:27 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843035/ [6843036] upgrade:octopus-x/parallel/{0-distro/rhel_8.5_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:16 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843036/ [6843037] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:57:31 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843037/ [6843038] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:10 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843038/ [6843039] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:06 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843039/ [6843041] upgrade:octopus-x/stress-split/{0-distro/rhel_8.5_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:15:31 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843041/ [6843042] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:38:27 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843042/ [6843043] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:19:18 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843043/ [6843044] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:04:23 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843044/ [6843045] upgrade:octopus-x/stress-split/{0-distro/rhel_8.5_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:51:31 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843045/ [6843046] upgrade:octopus-x/parallel/{0-distro/rhel_8.5_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:49 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843046/ [6843047] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:16:45 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843047/ [6843048] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:45:58 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843048/ [6843049] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:43:42 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843049/ [6843050] upgrade:octopus-x/parallel/{0-distro/rhel_8.5_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:07:58 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843050/ [6843051] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:46:47 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843051/ [6843052] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:48 info: http://pulpito.ceph.com/adking-2022-05-20_22:41:21-upgrade:octopus-x-wip-adk-testing-2022-05-19-2005-distro-basic-smithi/6843052/