2020-08-27T14:28:54.178 INFO:root:teuthology version: 1.0.0-8271c95 2020-08-27T14:28:54.180 INFO:teuthology.results:seed: '3291' 2020-08-27T14:36:24.178 INFO:root:teuthology version: 1.0.0-8271c95 2020-08-27T14:36:24.238 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-28T02:36:40.162 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-08-28T02:36:43.452 INFO:scrape:Found 16 jobs 2020-08-28T02:36:43.485 INFO:scrape:Found 2 distinct failure reasons 2020-08-28T02:36:43.486 INFO:scrape:Dead: b'2020-08-28T02:31:12.809 DEBUG:teuthology.exit:Finished running handlers' 2020-08-28T02:36:43.487 INFO:scrape:4 jobs: ['5380209', '5380217', '5380221', '5380216'] 2020-08-28T02:36:43.487 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_18.04}'] 2020-08-28T02:36:43.488 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2020-08-28T02:36:43.488 INFO:scrape: 2020-08-28T02:36:43.489 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=683) 2020-08-28T02:36:43.489 INFO:scrape:2 jobs: ['5380208', '5380213'] 2020-08-28T02:36:43.489 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_18.04}'] 2020-08-28T02:36:43.490 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2020-08-28T02:36:43.490 INFO:scrape: 2020-08-28T02:36:43.493 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/ failed: 0 dead: 3 running: 3 waiting: 0 queued: 0 passed: 10 Dead ================================================================= [5380208] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 11:48:35 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380208/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380208/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=3848d852c4ae42d19c517dfc1b50e7c7 psutil.NoSuchProcess process no longer exists (pid=683) [5380209] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380209/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380209/ [5380213] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} ----------------------------------------------------------------- time: 07:53:55 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380213/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380213/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=872ac2cfb7fd45fb8abd5087ac4e85ea psutil.NoSuchProcess process no longer exists (pid=15784) Running ================================================================= [5380216] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380216/ [5380217] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380217/ [5380221] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380221/ Pass ================================================================= [5380207] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 03:20:08 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380207/ [5380210] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 03:02:57 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380210/ [5380211] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 03:19:32 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380211/ [5380212] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:27:02 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380212/ [5380214] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 03:04:17 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380214/ [5380215] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 03:24:39 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380215/ [5380218] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 03:04:50 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380218/ [5380219] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04} time: 03:17:30 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380219/ [5380220] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04} time: 03:18:35 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380220/ [5380222] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04} time: 03:05:20 info: http://pulpito.ceph.com/teuthology-2020-08-27_14:23:03-upgrade:nautilus-x-master-distro-basic-smithi/5380222/