2021-05-22T14:22:27.415 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-22T14:22:27.416 INFO:teuthology.results:subset: '9/10' 2021-05-22T14:22:27.417 INFO:teuthology.results:seed: '3953' 2021-05-22T14:22:55.900 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-22T14:22:55.901 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-23T02:23:45.855 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-05-23T02:23:50.682 INFO:scrape:Found 35 jobs 2021-05-23T02:23:50.718 INFO:scrape:Found 2 distinct failure reasons 2021-05-23T02:23:50.719 INFO:scrape:Failure: Command failed on smithi185 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 2021-05-23T02:23:50.719 INFO:scrape:1 jobs: ['6128574'] 2021-05-23T02:23:50.720 INFO:scrape:suites: ['1-nautilus-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-pacific', '7-final-workload', 'mon_election/classic', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack'] 2021-05-23T02:23:50.721 INFO:scrape: 2021-05-23T02:23:50.721 INFO:scrape:Dead: b'2021-05-22T16:57:55.287 INFO:tasks.qemu.client.0.smithi176.stdout:[ 6480.274519] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.' 2021-05-23T02:23:50.722 INFO:scrape:1 jobs: ['6128581'] 2021-05-23T02:23:50.722 INFO:scrape:suites: ['1-nautilus-install/octopus', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-pacific', '7-final-workload', 'mon_election/connectivity', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack'] 2021-05-23T02:23:50.723 INFO:scrape: 2021-05-23T02:23:50.726 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/ failed: 1 dead: 0 running: 1 waiting: 0 queued: 0 passed: 33 Fail ================================================================= [6128574] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 01:33:16 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128574/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128574/ Command failed on smithi185 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' Running ================================================================= [6128581] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128581/ Pass ================================================================= [6128551] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:15:06 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128551/ [6128552] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 03:09:57 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128552/ [6128553] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:46:46 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128553/ [6128554] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 01:57:40 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128554/ [6128555] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:10:52 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128555/ [6128556] upgrade:octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:07:04 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128556/ [6128557] upgrade:octopus-x/parallel/{0-distro/centos_8.2_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:52 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128557/ [6128558] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:21:14 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128558/ [6128559] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 03:37:37 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128559/ [6128560] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 04:01:04 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128560/ [6128561] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:05:24 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128561/ [6128562] upgrade:octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:14:40 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128562/ [6128563] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:57:35 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128563/ [6128564] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:32 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128564/ [6128565] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:27:58 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128565/ [6128566] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 01:49:18 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128566/ [6128567] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:47:00 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128567/ [6128568] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_kubic_stable 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:11:04 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128568/ [6128569] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:50 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128569/ [6128570] upgrade:octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:24:25 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128570/ [6128571] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:04:46 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128571/ [6128572] upgrade:octopus-x/parallel/{0-distro/centos_8.2_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:15 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128572/ [6128573] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:20:41 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128573/ [6128575] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 05:01:29 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128575/ [6128576] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:09:36 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128576/ [6128577] upgrade:octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:16:27 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128577/ [6128578] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:00:13 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128578/ [6128579] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:29 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128579/ [6128580] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:27:50 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128580/ [6128582] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:47:43 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128582/ [6128583] upgrade:octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:59:32 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128583/ [6128584] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:29 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128584/ [6128585] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:25:21 info: http://pulpito.ceph.com/teuthology-2021-05-22_14:22:02-upgrade:octopus-x-pacific-distro-basic-smithi/6128585/