2020-05-11T14:24:04.425 INFO:root:teuthology version: 1.0.0-28c1ab5
2020-05-11T14:24:04.427 INFO:teuthology.results:seed: '4830'
2020-05-11T14:28:06.553 INFO:root:teuthology version: 1.0.0-28c1ab5
2020-05-11T14:28:06.557 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-05-11T16:36:08.934 INFO:teuthology.results:Tests finished! gathering results...
2020-05-11T16:36:08.993 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/
failed:  5
dead:    11
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[5046530]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-msgr2.yaml 8-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   01:26:19
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046530/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046530/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=6e128c5eb1b543d6bde92e5195044167

    psutil.NoSuchProcess process no longer exists (pid=28735)

[5046536]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046536/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046536/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=341599e6989c41d5a041baa5919eaafe

    Stale jobs detected, aborting.

[5046539]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/bluestore-bitmap.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:08:17
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046539/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046539/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=72a729f31b6e4fa88c9da625dc7427bd

    Command failed on smithi191 with status 1: 'sudo yum -y install ceph-
    immutable-object-cache'

[5046540]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046540/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046540/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=4de271a593014006801a1e5bb63df524

    Stale jobs detected, aborting.

[5046544]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046544/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046544/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=f5f3895468364d21a0af7ebdd906d80b

    Stale jobs detected, aborting.



Dead
=================================================================
[5046529]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046529/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046529/

[5046531]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/bluestore-bitmap.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046531/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046531/

[5046532]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046532/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046532/

[5046533]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046533/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046533/

[5046534]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-msgr2.yaml 8-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046534/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046534/

[5046535]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/filestore-xfs.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046535/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046535/

[5046537]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046537/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046537/

[5046538]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-msgr2.yaml 8-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046538/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046538/

[5046541]  upgrade:nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-octopus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046541/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046541/

[5046542]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-msgr2.yaml 8-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046542/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046542/

[5046543]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/filestore-xfs.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 6-octopus.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046543/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-11_14:23:02-upgrade:nautilus-x-octopus-distro-basic-smithi/5046543/
2020-05-11T16:36:09.116 INFO:teuthology.results:starting coverage generation
2020-05-11T16:36:09.123 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'