2021-04-25T03:46:25.404 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-25T03:46:25.405 INFO:teuthology.results:subset: '1/10' 2021-04-25T03:46:25.405 INFO:teuthology.results:seed: '5374' 2021-04-25T06:19:22.920 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-25T06:19:22.950 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-04-25T10:10:55.651 INFO:teuthology.results:Tests finished! gathering results... 2021-04-25T10:10:58.836 INFO:scrape:Found 29 jobs 2021-04-25T10:10:59.204 INFO:scrape:Found 1 distinct failure reasons 2021-04-25T10:10:59.204 INFO:scrape:Failure: "2021-04-25T04:50:21.796606+0000 mon.a (mon.0) 712 : cluster [WRN] Health check failed: 1/464 objects unfound (0.216%) (OBJECT_UNFOUND)" in cluster log 2021-04-25T10:10:59.205 INFO:scrape:1 jobs: ['6071134'] 2021-04-25T10:10:59.205 INFO:scrape:suites: ['1-ceph-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-workload/{rgw_ragweed_prepare}', '3-upgrade-sequence/upgrade-all', '4-pacific', '5-final-workload/{rgw', 'centos_latest}', 'rgw_ragweed_check}', 'start}', 'upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack'] 2021-04-25T10:10:59.205 INFO:scrape: 2021-04-25T10:10:59.208 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 28 Fail ================================================================= [6071134] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} ----------------------------------------------------------------- time: 00:19:15 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071134/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071134/ "2021-04-25T04:50:21.796606+0000 mon.a (mon.0) 712 : cluster [WRN] Health check failed: 1/464 objects unfound (0.216%) (OBJECT_UNFOUND)" in cluster log Pass ================================================================= [6071108] upgrade:octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:02:50 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071108/ [6071109] upgrade:octopus-x/parallel/{0-distro/centos_8.2_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:46 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071109/ [6071110] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:19:50 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071110/ [6071111] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 03:26:01 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071111/ [6071112] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 05:16:39 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071112/ [6071113] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:11:57 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071113/ [6071114] upgrade:octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:14:15 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071114/ [6071115] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 02:08:12 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071115/ [6071116] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:24 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071116/ [6071117] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:26:37 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071117/ [6071120] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:30:42 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071120/ [6071122] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 01:44:34 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071122/ [6071125] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:38:17 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071125/ [6071127] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:53:52 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071127/ [6071129] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:35 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071129/ [6071131] upgrade:octopus-x/stress-split/{0-distro/rhel_8.3_kubic_stable 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:10:24 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071131/ [6071133] upgrade:octopus-x/parallel/{0-distro/centos_8.2_kubic_stable 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:05:57 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071133/ [6071136] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} time: 01:29:24 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071136/ [6071138] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} time: 05:18:05 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071138/ [6071140] upgrade:octopus-x/stress-split/{0-distro/ubuntu_18.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:55:05 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071140/ [6071142] upgrade:octopus-x/parallel/{0-distro/rhel_8.3_kubic_stable 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:15:33 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071142/ [6071144] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} time: 02:09:35 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071144/ [6071146] upgrade:octopus-x/parallel/{0-distro/ubuntu_18.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:06:52 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071146/ [6071148] upgrade:octopus-x/parallel-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-workload/{rgw_ragweed_prepare} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{rgw rgw_ragweed_check} centos_latest} time: 00:26:38 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071148/ [6071150] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-nautilus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-pacific 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} time: 03:27:46 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071150/ [6071152] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 02:44:25 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071152/ [6071154] upgrade:octopus-x/stress-split/{0-distro/centos_8.2_kubic_stable 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} time: 01:58:25 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071154/ [6071156] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} time: 01:09:37 info: http://pulpito.ceph.com/teuthology-2021-04-24_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/6071156/