2022-02-25T14:31:15.900 INFO:root:teuthology version: 1.1.0-5825e8d0 2022-02-25T14:31:15.902 INFO:teuthology.results:subset: '8/10' 2022-02-25T14:31:15.902 INFO:teuthology.results:seed: '9554' 2022-02-25T15:17:56.824 INFO:root:teuthology version: 1.1.0-5825e8d0 2022-02-25T15:17:56.866 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-25T18:52:12.756 INFO:teuthology.results:Tests finished! gathering results... 2022-02-25T18:52:16.401 INFO:scrape:Found 32 jobs 2022-02-25T18:52:16.562 WARNING:scrape:Not-understood crash indication 2022-02-25T15:27:46.848 INFO:tasks.rgw.c2.client.0.smithi160.stderr:daemon-helper: command crashed with signal 11 2022-02-25T18:52:16.564 WARNING:scrape:Not-understood crash indication 2022-02-25T18:52:16.586 WARNING:scrape:Not-understood crash indication 2022-02-25T15:27:46.848 INFO:tasks.rgw.c2.client.0.smithi160.stderr:daemon-helper: command crashed with signal 11 2022-02-25T18:52:16.587 WARNING:scrape:Not-understood crash indication 2022-02-25T18:52:16.605 WARNING:scrape:Not-understood crash indication 2022-02-25T15:27:46.848 INFO:tasks.rgw.c2.client.0.smithi160.stderr:daemon-helper: command crashed with signal 11 2022-02-25T18:52:16.606 WARNING:scrape:Not-understood crash indication 2022-02-25T18:52:16.622 WARNING:scrape:Not-understood crash indication 2022-02-25T15:27:46.848 INFO:tasks.rgw.c2.client.0.smithi160.stderr:daemon-helper: command crashed with signal 11 2022-02-25T18:52:16.623 WARNING:scrape:Not-understood crash indication 2022-02-25T18:52:16.766 WARNING:scrape:Not-understood crash indication 2022-02-25T15:05:11.776 INFO:tasks.rgw.c2.client.0.smithi176.stderr:daemon-helper: command crashed with signal 11 2022-02-25T18:52:16.767 WARNING:scrape:Not-understood crash indication 2022-02-25T18:52:16.792 WARNING:scrape:Not-understood crash indication 2022-02-25T15:05:11.776 INFO:tasks.rgw.c2.client.0.smithi176.stderr:daemon-helper: command crashed with signal 11 2022-02-25T18:52:16.794 WARNING:scrape:Not-understood crash indication 2022-02-25T18:52:16.796 INFO:scrape:Found 2 distinct failure reasons 2022-02-25T18:52:16.797 INFO:scrape:Failure: Command failed on smithi184 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' 2022-02-25T18:52:16.797 INFO:scrape:4 jobs: ['6705794', '6705787', '6705803', '6705810'] 2022-02-25T18:52:16.798 INFO:scrape:suites intersection: ['1-ceph-install/octopus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2022-02-25T18:52:16.799 INFO:scrape:suites union: ['1-ceph-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack'] 2022-02-25T18:52:16.799 INFO:scrape: 2022-02-25T18:52:16.800 INFO:scrape:Failure: rgw multisite test failures 2022-02-25T18:52:16.800 INFO:scrape:2 jobs: ['6705800', '6705784'] 2022-02-25T18:52:16.801 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-02-25T18:52:16.801 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2022-02-25T18:52:16.801 INFO:scrape: 2022-02-25T18:52:16.804 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 26 Fail ================================================================= [6705784] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 03:07:51 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705784/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d0a4fbf7e9b45d58dbae97588a8ef92 rgw multisite test failures [6705787] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:24:12 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705787/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9ea00785d54401b9daf91bdea4adac7 Command failed on smithi178 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6705794] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:53:19 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705794/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705794/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a5f4d005d3b4232bde8ef9fde01c1ef Command failed on smithi184 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6705800] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 03:06:46 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705800/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53e33d90657443609d0bfe016277fcd8 rgw multisite test failures [6705803] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:30:46 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705803/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57eea0bd31c64e0284be4b2c6eb1f351 Command failed on smithi198 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' [6705810] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:52:38 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705810/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6659f8240c154e3c9709b0bbe42ae297 Command failed on smithi135 with status 2: 'cd /home/ubuntu/cephtest/ragweed && ./bootstrap' Pass ================================================================= [6705783] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:53 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705783/ [6705785] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:11:50 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705785/ [6705786] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 04:06:23 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705786/ [6705788] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:25:51 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705788/ [6705789] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:20 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705789/ [6705790] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:01:54 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705790/ [6705791] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:05 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705791/ [6705792] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:18:31 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705792/ [6705793] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 01:43:40 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705793/ [6705795] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:46 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705795/ [6705796] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:39:05 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705796/ [6705797] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:25 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705797/ [6705798] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:03:56 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705798/ [6705799] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:19 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705799/ [6705801] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:30:05 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705801/ [6705802] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 02:16:52 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705802/ [6705804] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:13:43 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705804/ [6705805] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:01:26 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705805/ [6705806] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:08:37 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705806/ [6705807] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:39 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705807/ [6705808] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 01:56:56 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705808/ [6705809] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:36:41 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705809/ [6705811] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:17:35 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705811/ [6705812] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:38:25 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705812/ [6705813] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:08:26 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705813/ [6705814] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:21:16 info: http://pulpito.ceph.com/teuthology-2022-02-25_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6705814/