2020-07-09T02:56:19.933 INFO:root:teuthology version: 1.0.0-22d0a87 2020-07-09T02:56:19.934 INFO:teuthology.results:subset: '27/9999' 2020-07-09T02:56:19.935 INFO:teuthology.results:seed: '4926' 2020-07-09T03:41:19.714 INFO:root:teuthology version: 1.0.0-22d0a87 2020-07-09T03:41:19.743 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-07-09T07:09:24.375 INFO:teuthology.results:Tests finished! gathering results... 2020-07-09T07:09:27.889 INFO:scrape:Found 43 jobs 2020-07-09T07:09:52.578 INFO:scrape:Found 6 distinct failure reasons 2020-07-09T07:09:52.580 INFO:scrape:Failure: Command failed on smithi031 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' 2020-07-09T07:09:52.581 INFO:scrape:20 jobs: ['5206435', '5206401', '5206407', '5206430', '5206433', '5206431', '5206422', '5206398', '5206414', '5206427', '5206423', '5206436', '5206412', '5206420', '5206406', '5206396', '5206400', '5206409', '5206404', '5206416'] 2020-07-09T07:09:52.582 INFO:scrape:suites intersection: ['conf/{client', 'log-config', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.582 INFO:scrape:suites union: ['clusters/1-mds-1-client', 'clusters/1-mds-2-client', 'conf/{client', 'inline/no', 'inline/yes', 'k-testing}', 'kcephfs/cephfs/{begin', 'kcephfs/mixed-clients/{begin', 'kcephfs/thrash/{begin', 'kclient-overrides/{distro/stock/{k-stock', 'kclient-overrides/{distro/testing/{flavor/centos_latest', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/kclient_workunit_direct_io}', 'tasks/kclient_workunit_kernel_untar_build}', 'tasks/kclient_workunit_o_trunc}', 'tasks/kclient_workunit_snaps}', 'tasks/kclient_workunit_suites_ffsb}', 'tasks/kclient_workunit_suites_fsx}', 'tasks/kclient_workunit_suites_fsync}', 'tasks/kclient_workunit_suites_iozone}', 'tasks/kclient_workunit_suites_pjd}', 'tasks/kclient_workunit_trivial_sync}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'thrash-health-whitelist', 'thrashers/default', 'thrashers/mds', 'thrashers/mon', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workloads/kclient_workunit_suites_ffsb}', 'workloads/kclient_workunit_suites_iozone}'] 2020-07-09T07:09:52.583 INFO:scrape: 2020-07-09T07:09:52.583 INFO:scrape:Failure: "2020-07-09T03:35:02.151958+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log 2020-07-09T07:09:52.584 INFO:scrape:7 jobs: ['5206418', '5206428', '5206415', '5206424', '5206429', '5206410', '5206437'] 2020-07-09T07:09:52.584 INFO:scrape:suites intersection: ['conf/{client', 'log-config', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.585 INFO:scrape:suites union: ['clusters/1-mds-1-client', 'clusters/1-mds-2-client', 'clusters/1-mds-4-client', 'conf/{client', 'inline/no', 'inline/yes', 'k-testing}', 'kcephfs/cephfs/{begin', 'kcephfs/mixed-clients/{begin', 'kcephfs/recovery/{begin', 'kcephfs/thrash/{begin', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/filestore-xfs', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/kclient_workunit_misc}', 'tasks/kclient_workunit_suites_dbench}', 'tasks/kclient_workunit_suites_fsstress}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mds-full}', 'thrash-health-whitelist', 'thrashers/mds', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workloads/kclient_workunit_suites_ffsb}'] 2020-07-09T07:09:52.585 INFO:scrape: 2020-07-09T07:09:52.586 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2020-07-09T07:09:52.586 INFO:scrape:1 jobs: ['5206413'] 2020-07-09T07:09:52.587 INFO:scrape:suites: ['clusters/1-mds-4-client', 'conf/{client', 'k-testing}', 'kcephfs/recovery/{begin', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.587 INFO:scrape: 2020-07-09T07:09:52.588 INFO:scrape:Failure: Command failed on smithi150 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 2020-07-09T07:09:52.588 INFO:scrape:5 jobs: ['5206411', '5206403', '5206426', '5206434', '5206419'] 2020-07-09T07:09:52.589 INFO:scrape:suites intersection: ['clusters/1-mds-4-client', 'conf/{client', 'k-testing}', 'kcephfs/recovery/{begin', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.590 INFO:scrape:suites union: ['clusters/1-mds-4-client', 'conf/{client', 'k-testing}', 'kcephfs/recovery/{begin', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/auto-repair}', 'tasks/client-recovery}', 'tasks/failover}', 'tasks/mds-flush}', 'tasks/sessionmap}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.590 INFO:scrape: 2020-07-09T07:09:52.590 INFO:scrape:Failure: "2020-07-09T04:40:25.003098+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi150:0 (5510), after waiting 48.1367 seconds during MDS startup" in cluster log 2020-07-09T07:09:52.591 INFO:scrape:1 jobs: ['5206438'] 2020-07-09T07:09:52.591 INFO:scrape:suites: ['clusters/1-mds-4-client', 'conf/{client', 'k-testing}', 'kcephfs/recovery/{begin', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}}', 'objectstore-ec/filestore-xfs', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.592 INFO:scrape: 2020-07-09T07:09:52.592 INFO:scrape:Failure: Test failure: test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair) 2020-07-09T07:09:52.593 INFO:scrape:1 jobs: ['5206425'] 2020-07-09T07:09:52.594 INFO:scrape:suites: ['clusters/1-mds-4-client', 'conf/{client', 'kcephfs/recovery/{begin', 'kclient/{mount', 'log-config', 'mds', 'mon', 'ms-die-on-skipped}}', 'objectstore-ec/filestore-xfs', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/journal-repair}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-07-09T07:09:52.594 INFO:scrape: 2020-07-09T07:09:52.598 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/ failed: 35 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [5206396] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:47 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206396/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206396/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=2108628950fd4ff88692a53f2d942a86 Command failed on smithi045 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206398] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:18:43 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206398/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206398/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=2cc196242f674a4a9a7194d9d9a4c392 Command failed on smithi157 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206400] kcephfs/thrash/{begin clusters/1-mds-1-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts thrash-health-whitelist whitelist_health whitelist_wrongly_marked_down} thrashers/default workloads/kclient_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:14:33 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206400/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206400/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=01bd304213b44b8299dd254a2f6e8558 Command failed on smithi121 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206401] kcephfs/mixed-clients/{begin clusters/1-mds-2-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206401/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206401/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=9ec49fecf0d44046a6c12db1b0524e4e Command failed on smithi179 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206403] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} ----------------------------------------------------------------- time: 00:03:48 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206403/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206403/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=ef5e30dc875648edb1a29cbf70847147 Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl- devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr- devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel' [5206404] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_direct_io} ----------------------------------------------------------------- time: 00:13:56 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206404/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206404/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=18df9ed368174f108dacfdd2c391f847 Command failed on smithi117 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206406] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_kernel_untar_build} ----------------------------------------------------------------- time: 00:19:03 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206406/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206406/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=72e31735bba04093ac75ac0c3c5c3c1e Command failed on smithi043 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206407] kcephfs/thrash/{begin clusters/1-mds-1-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts thrash-health-whitelist whitelist_health whitelist_wrongly_marked_down} thrashers/mds workloads/kclient_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:18:19 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206407/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206407/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=8564d049f2a04ef79b3b736889f3a2a2 Command failed on smithi042 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206409] kcephfs/mixed-clients/{begin clusters/1-mds-2-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206409/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206409/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=cb715ba7b10f4814a1c72bf8e83aee91 Command failed on smithi189 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206410] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_misc} ----------------------------------------------------------------- time: 00:42:20 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206410/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206410/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=70c6ede6e5114e7cba7480e6c456a3d4 "2020-07-09T03:22:52.996765+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [5206411] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} ----------------------------------------------------------------- time: 00:03:30 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206411/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206411/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=5c744b4b247b42528fb5742a9640f2e4 Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl- devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr- devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel' [5206412] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_o_trunc} ----------------------------------------------------------------- time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206412/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206412/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=25f36ba8bf344c339c92792d9b7b63d3 Command failed on smithi136 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206413] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/damage} ----------------------------------------------------------------- time: 00:11:41 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206413/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206413/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=58cf41b1cdf04cc9a19a761dcebddc08 Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) [5206414] kcephfs/thrash/{begin clusters/1-mds-1-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts thrash-health-whitelist whitelist_health whitelist_wrongly_marked_down} thrashers/mon workloads/kclient_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:14:29 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206414/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206414/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=ba1e1db1a6e2465a82e4639dabff6cde Command failed on smithi092 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206415] kcephfs/mixed-clients/{begin clusters/1-mds-2-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:42:34 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206415/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206415/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=25a502b97bd3446aa0068214df37f825 "2020-07-09T03:39:57.012270+0000 mon.b (mon.0) 141 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [5206416] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_snaps} ----------------------------------------------------------------- time: 00:18:48 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206416/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206416/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=dff6af9cfe5643c88f58ee6f5c288e4d Command failed on smithi045 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206418] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:42:26 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206418/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206418/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=d6098f30c43f4696a89d347a53b7f575 "2020-07-09T03:35:02.151958+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [5206419] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/failover} ----------------------------------------------------------------- time: 00:03:49 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206419/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206419/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e24bf1dcd2fb4860a776e6000dee774a Command failed on smithi052 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl- devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr- devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel' [5206420] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206420/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206420/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=3dceafae686c472ca6792d0ff2e4eea3 Command failed on smithi107 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206422] kcephfs/thrash/{begin clusters/1-mds-1-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts thrash-health-whitelist whitelist_health whitelist_wrongly_marked_down} thrashers/default workloads/kclient_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:14:50 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206422/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206422/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=bc7dd789a7da403a89d618f5de1be6d7 Command failed on smithi050 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206423] kcephfs/mixed-clients/{begin clusters/1-mds-2-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:18:47 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206423/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206423/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=b3e848b8dcd543ff984d713aa56fd65d Command failed on smithi152 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206424] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:47:11 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206424/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206424/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=453de318da33433298c78d0ac9ab8924 "2020-07-09T03:38:07.645690+0000 mon.b (mon.0) 144 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [5206425] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} ----------------------------------------------------------------- time: 00:23:44 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206425/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206425/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=202d41cbfe5446d6a8edebd68cbd2bb5 Test failure: test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair) [5206426] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} ----------------------------------------------------------------- time: 00:03:39 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206426/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206426/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=c24598c3738d44c5a9fda4ffbe7eccc4 Command failed on smithi107 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl- devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr- devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel' [5206427] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206427/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206427/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=b28822aeecb64257be0f19ab92b3e9d3 Command failed on smithi148 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206428] kcephfs/thrash/{begin clusters/1-mds-1-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts thrash-health-whitelist whitelist_health whitelist_wrongly_marked_down} thrashers/mds workloads/kclient_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:42:36 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206428/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206428/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=532da41adfb14d72aae15ea3d581e12f "2020-07-09T03:45:56.674867+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log [5206429] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} ----------------------------------------------------------------- time: 00:43:24 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206429/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206429/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=5697bd0c3ca74f398bab8cfff2ebfb38 "2020-07-09T03:48:33.138626+0000 mon.a (mon.0) 163 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [5206430] kcephfs/mixed-clients/{begin clusters/1-mds-2-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:14:51 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206430/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206430/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=0fc0072136a14c04a7b84376c6d7f87b Command failed on smithi041 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206431] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_fsync} ----------------------------------------------------------------- time: 00:13:48 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206431/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206431/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=43bfacbc01a2470ea55a5f089dc9d5b6 Command failed on smithi141 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206433] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:18:25 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206433/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206433/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e659785892e945408441f2bbf1e502ae Command failed on smithi187 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206434] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} ----------------------------------------------------------------- time: 00:03:31 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206434/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206434/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=da86c4e1261644fdbda4a64c5db2fd54 Command failed on smithi164 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl- devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr- devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel' [5206435] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:26 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206435/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206435/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=32b12c1a0273428687cbbd8801ae9f33 Command failed on smithi031 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206436] kcephfs/thrash/{begin clusters/1-mds-1-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts thrash-health-whitelist whitelist_health whitelist_wrongly_marked_down} thrashers/mon workloads/kclient_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:18:32 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206436/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206436/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=d759fef2cd64484fb13f4ee0e8224768 Command failed on smithi092 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout baa1ea6a9656c3db06c66032fa80b476721947ba' [5206437] kcephfs/mixed-clients/{begin clusters/1-mds-2-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:42:46 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206437/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206437/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=829e2637a4ef413381e232ddaea6f5f3 "2020-07-09T03:56:46.694178+0000 mon.a (mon.0) 232 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [5206438] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/strays} ----------------------------------------------------------------- time: 00:27:35 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206438/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206438/ "2020-07-09T04:40:25.003098+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi150:0 (5510), after waiting 48.1367 seconds during MDS startup" in cluster log Pass ================================================================= [5206397] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/strays} time: 00:25:21 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206397/ [5206399] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/volume-client} time: 00:31:05 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206399/ [5206402] kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/yes kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/acls-kernel-client} time: 00:12:06 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206402/ [5206405] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} time: 00:12:09 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206405/ [5206408] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} time: 00:21:52 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206408/ [5206417] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} time: 00:33:32 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206417/ [5206421] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} time: 00:15:04 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206421/ [5206432] kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:19:59 info: http://pulpito.ceph.com/teuthology-2020-07-07_10:25:02-kcephfs-master-testing-basic-smithi/5206432/