2020-09-09T18:29:23.884 INFO:root:teuthology version: 1.0.0-55bd2dc 2020-09-09T18:29:23.884 INFO:teuthology.results:seed: '9408' 2020-09-09T19:15:47.669 INFO:root:teuthology version: 1.0.0-55bd2dc 2020-09-09T19:15:47.675 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-09-10T07:15:59.540 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-09-10T07:16:02.115 INFO:scrape:Found 16 jobs 2020-09-10T07:16:03.112 INFO:scrape:Found 4 distinct failure reasons 2020-09-10T07:16:03.113 INFO:scrape:Failure: timeout expired in wait_for_all_osds_up 2020-09-10T07:16:03.113 INFO:scrape:2 jobs: ['5419242', '5419250'] 2020-09-10T07:16:03.114 INFO:scrape:suites intersection: ['1-ceph-install/nautilus', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-octopus', '5-final-workload/{blogbench', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack'] 2020-09-10T07:16:03.114 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-octopus', '5-final-workload/{blogbench', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack'] 2020-09-10T07:16:03.114 INFO:scrape: 2020-09-10T07:16:03.115 INFO:scrape:Dead: b'2020-09-10T07:05:51.965 DEBUG:teuthology.exit:Finished running handlers' 2020-09-10T07:16:03.115 INFO:scrape:4 jobs: ['5419248', '5419239', '5419244', '5419247'] 2020-09-10T07:16:03.115 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_18.04}'] 2020-09-10T07:16:03.116 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2020-09-10T07:16:03.116 INFO:scrape: 2020-09-10T07:16:03.116 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=3129) 2020-09-10T07:16:03.116 INFO:scrape:1 jobs: ['5419240'] 2020-09-10T07:16:03.117 INFO:scrape:suites: ['1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-octopus', '7-final-workload', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2020-09-10T07:16:03.117 INFO:scrape: 2020-09-10T07:16:03.117 INFO:scrape:Dead: b'2020-09-10T07:16:02.897 INFO:teuthology.orchestra.run.smithi058.stderr:2020-09-10T07:16:02.895+0000 7fb92f59e700 -1 WARNING: all dangerous and experimental features are enabled.' 2020-09-10T07:16:03.118 INFO:scrape:1 jobs: ['5419252'] 2020-09-10T07:16:03.118 INFO:scrape:suites: ['1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-octopus', '7-final-workload', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack'] 2020-09-10T07:16:03.118 INFO:scrape: 2020-09-10T07:16:03.120 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/ failed: 2 dead: 5 running: 1 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [5419242] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:22:57 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419242/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419242/ sentry: https://sentry.ceph.com/sepia/teuthology/?query=aa19f8d1b63d4c5a81759e35b17d2c6a timeout expired in wait_for_all_osds_up [5419250] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} ----------------------------------------------------------------- time: 00:22:40 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419250/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419250/ sentry: https://sentry.ceph.com/sepia/teuthology/?query=6f7c60b281864794b8eac8fde8002ab4 timeout expired in wait_for_all_osds_up Dead ================================================================= [5419239] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419239/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419239/ [5419240] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 11:47:38 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419240/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419240/ sentry: https://sentry.ceph.com/sepia/teuthology/?query=9814ecdc3d854011a5e51335a50b4348 psutil.NoSuchProcess process no longer exists (pid=3129) [5419244] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419244/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419244/ [5419247] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419247/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419247/ [5419248] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419248/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419248/ Running ================================================================= [5419252] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419252/ Pass ================================================================= [5419238] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 02:30:28 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419238/ [5419241] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 02:13:50 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419241/ [5419243] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:03:57 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419243/ [5419245] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 02:14:08 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419245/ [5419246] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 02:32:02 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419246/ [5419249] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 02:14:19 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419249/ [5419251] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:21:13 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419251/ [5419253] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 02:14:23 info: http://pulpito.ceph.com/teuthology-2020-09-09_14:23:02-upgrade:nautilus-x-master-distro-basic-smithi/5419253/