2020-05-23T21:50:33.348 INFO:root:teuthology version: 1.0.0-4413d31 2020-05-23T21:50:33.350 INFO:teuthology.results:subset: '1/9999' 2020-05-23T21:50:33.350 INFO:teuthology.results:seed: '8721' 2020-05-23T22:09:54.843 INFO:root:teuthology version: 1.0.0-4413d31 2020-05-23T22:09:54.871 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-05-24T10:10:09.523 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-05-24T10:10:09.610 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/ failed: 2 dead: 13 running: 1 waiting: 0 queued: 0 passed: 27 Fail ================================================================= [5066823] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-full.yaml} ----------------------------------------------------------------- time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066823/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066823/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=2dfdeece11774eccb0472b9f9fa2de8f Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) [5066827] kcephfs/thrash/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml thrash-health-whitelist.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrashers/mds.yaml workloads/kclient_workunit_suites_ffsb.yaml} ----------------------------------------------------------------- time: 00:28:36 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066827/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066827/ "2020-05-23T22:27:25.363523+0000 mds.b (mds.0) 2 : cluster [WRN] client.4400 isn't responding to mclientcaps(revoke), ino 0x1000000037f pending pAsLsXsFsc issued pAsLsXsFsc, sent 60.556263 seconds ago" in cluster log Dead ================================================================= [5066801] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-limits.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066801/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066801/ [5066804] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_misc.yaml} ----------------------------------------------------------------- time: 01:09:53 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066804/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066804/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=3388c6c3e67947d7a088967396926e45 SSH connection to smithi077 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [5066807] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/damage.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066807/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066807/ [5066809] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/data-scan.yaml} ----------------------------------------------------------------- time: 01:02:21 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066809/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066809/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=9a0632ee752a4cdeb53233c012e18544 Test failure: test_rebuild_linkage (tasks.cephfs.test_data_scan.TestDataScan) [5066811] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/failover.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066811/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066811/ [5066812] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_dbench.yaml} ----------------------------------------------------------------- time: 01:10:47 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066812/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066812/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=4b2df0cf0ef04c2cac6bc54c46fde229 SSH connection to smithi040 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [5066813] kcephfs/thrash/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml thrash-health-whitelist.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrashers/mon.yaml workloads/kclient_workunit_suites_ffsb.yaml} ----------------------------------------------------------------- time: 01:04:25 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066813/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066813/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=f86504332a6a4209ac56f69ec95fb240 SSH connection to smithi092 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [5066814] kcephfs/mixed-clients/{begin.yaml clusters/1-mds-2-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient-overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kernel_cfuse_workunits_dbench_iozone.yaml} ----------------------------------------------------------------- time: 01:09:26 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066814/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066814/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e690a6617a4546ceafdc825b620d3f78 SSH connection to smithi014 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf' [5066815] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/forward-scrub.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066815/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066815/ [5066817] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/journal-repair.yaml} ----------------------------------------------------------------- time: 00:50:24 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066817/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066817/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e073e08dfb374812b03a5023cf131958 Test failure: test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair), test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair) [5066822] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_fsx.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066822/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066822/ [5066833] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/volume-client.yaml} ----------------------------------------------------------------- time: 00:51:36 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066833/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066833/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=2a99b732a7ad454dbbd2a7f6a0dcd9d0 Test failure: test_idempotency (tasks.cephfs.test_volume_client.TestVolumeClient) [5066834] kcephfs/thrash/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml thrash-health-whitelist.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrashers/mon.yaml workloads/kclient_workunit_suites_iozone.yaml} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066834/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066834/ Running ================================================================= [5066835] kcephfs/mixed-clients/{begin.yaml clusters/1-mds-2-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient-overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066835/ Pass ================================================================= [5066797] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/backtrace.yaml} time: 00:15:02 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066797/ [5066798] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_direct_io.yaml} time: 00:15:51 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066798/ [5066799] kcephfs/thrash/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml thrash-health-whitelist.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrashers/default.yaml workloads/kclient_workunit_suites_ffsb.yaml} time: 00:51:04 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066799/ [5066800] kcephfs/mixed-clients/{begin.yaml clusters/1-mds-2-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient-overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kernel_cfuse_workunits_dbench_iozone.yaml} time: 01:01:30 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066800/ [5066802] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_kernel_untar_build.yaml} time: 00:48:52 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066802/ [5066803] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-recovery.yaml} time: 00:38:39 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066803/ [5066805] kcephfs/thrash/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml thrash-health-whitelist.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrashers/mds.yaml workloads/kclient_workunit_suites_iozone.yaml} time: 00:17:29 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066805/ [5066806] kcephfs/mixed-clients/{begin.yaml clusters/1-mds-2-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient-overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} time: 00:45:39 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066806/ [5066808] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_o_trunc.yaml} time: 00:25:33 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066808/ [5066810] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_snaps.yaml} time: 00:42:21 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066810/ [5066816] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_ffsb.yaml} time: 01:04:55 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066816/ [5066818] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_fsstress.yaml} time: 00:16:52 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066818/ [5066819] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-flush.yaml} time: 00:19:55 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066819/ [5066820] kcephfs/thrash/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml thrash-health-whitelist.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrashers/default.yaml workloads/kclient_workunit_suites_iozone.yaml} time: 00:25:06 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066820/ [5066821] kcephfs/mixed-clients/{begin.yaml clusters/1-mds-2-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient-overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} time: 00:34:25 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066821/ [5066824] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_fsync.yaml} time: 00:16:06 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066824/ [5066825] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/pool-perm.yaml} time: 00:11:52 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066825/ [5066826] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_iozone.yaml} time: 00:15:38 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066826/ [5066828] kcephfs/mixed-clients/{begin.yaml clusters/1-mds-2-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient-overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kernel_cfuse_workunits_dbench_iozone.yaml} time: 00:48:57 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066828/ [5066829] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/sessionmap.yaml} time: 00:20:48 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066829/ [5066830] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_suites_pjd.yaml} time: 00:13:29 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066830/ [5066831] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/strays.yaml} time: 00:29:20 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066831/ [5066832] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_trivial_sync.yaml} time: 00:15:52 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066832/ [5066836] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/acls-kernel-client.yaml} time: 00:14:00 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066836/ [5066837] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/auto-repair.yaml} time: 00:18:20 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066837/ [5066838] kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/backtrace.yaml} time: 00:14:38 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066838/ [5066839] kcephfs/cephfs/{begin.yaml clusters/1-mds-1-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/kclient_workunit_direct_io.yaml} time: 00:12:05 info: http://pulpito.ceph.com/teuthology-2020-05-18_10:25:03-kcephfs-master-testing-basic-smithi/5066839/ 2020-05-24T10:10:09.773 INFO:teuthology.results:starting coverage generation 2020-05-24T10:10:09.778 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results generate_coverage(archive_dir, name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage archive_dir, File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'