2020-05-28T14:23:55.098 INFO:root:teuthology version: 1.0.0-5ae3303
2020-05-28T14:23:55.100 INFO:teuthology.results:seed: '3770'
2020-05-28T14:30:13.667 INFO:root:teuthology version: 1.0.0-5ae3303
2020-05-28T14:30:13.670 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-05-28T20:27:21.347 INFO:teuthology.results:Tests finished! gathering results...
2020-05-28T20:27:21.408 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  12


Fail
=================================================================
[5098759]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   02:23:03
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098759/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098759/

    "2020-05-28T15:19:48.029986+0000 mds.a (mds.0) 5 : cluster [WRN] evicting
    unresponsive client smithi167: (4851), after waiting 48.1829 seconds during
    MDS startup" in cluster log

[5098760]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   01:11:12
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098760/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098760/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=900535563d2842988f2c9ea4418027c7

    Command failed on smithi087 with status 1: 'cd
    /home/ubuntu/cephtest/ragweed && ./bootstrap'

[5098767]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   02:30:02
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098767/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098767/

    "2020-05-28T15:39:03.917043+0000 mds.a (mds.0) 5 : cluster [WRN] evicting
    unresponsive client smithi139: (54293), after waiting 49.6383 seconds
    during MDS startup" in cluster log

[5098768]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   02:27:45
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098768/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098768/

    "2020-05-28T15:30:02.491419+0000 mds.a (mds.0) 5 : cluster [WRN] evicting
    unresponsive client smithi152: (14276), after waiting 48.3232 seconds
    during MDS startup" in cluster log



Pass
=================================================================
[5098756] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
time:   02:32:27
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098756/

[5098757] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
time:   03:31:20
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098757/

[5098758] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04}
time:   02:16:19
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098758/

[5098761] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
time:   03:18:32
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098761/

[5098762] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04}
time:   01:43:52
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098762/

[5098763] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
time:   02:24:51
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098763/

[5098764] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
time:   02:29:07
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098764/

[5098765] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
time:   03:50:24
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098765/

[5098766] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04}
time:   02:00:51
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098766/

[5098769] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
time:   03:19:13
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098769/

[5098770] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04}
time:   01:54:14
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098770/

[5098771] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
time:   02:25:30
info:   http://pulpito.ceph.com/teuthology-2020-05-28_14:23:04-upgrade:nautilus-x-master-distro-basic-smithi/5098771/
2020-05-28T20:27:21.483 INFO:teuthology.results:starting coverage generation
2020-05-28T20:27:21.490 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'