2020-08-30T15:41:23.191 INFO:root:teuthology version: 1.0.0-018d708 2020-08-30T15:41:23.193 INFO:teuthology.results:subset: '18/9999' 2020-08-30T15:41:23.193 INFO:teuthology.results:seed: '4896' 2020-08-30T15:53:11.688 INFO:root:teuthology version: 1.0.0-018d708 2020-08-30T15:53:11.727 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-30T19:08:16.565 INFO:teuthology.results:Tests finished! gathering results... 2020-08-30T19:08:19.730 INFO:scrape:Found 24 jobs 2020-08-30T19:08:20.747 INFO:scrape:Found 3 distinct failure reasons 2020-08-30T19:08:20.749 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2020-08-30T19:08:20.749 INFO:scrape:1 jobs: ['5383208'] 2020-08-30T19:08:20.750 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}'] 2020-08-30T19:08:20.750 INFO:scrape: 2020-08-30T19:08:20.751 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/blogbench.sh 2020-08-30T19:08:20.751 INFO:scrape:1 jobs: ['5383194'] 2020-08-30T19:08:20.751 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-08-30T19:08:20.752 INFO:scrape: 2020-08-30T19:08:20.752 INFO:scrape:Failure: Command failed on smithi162 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 2020-08-30T19:08:20.753 INFO:scrape:1 jobs: ['5383204'] 2020-08-30T19:08:20.753 INFO:scrape:suites: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress', 'validater/lockdep}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}'] 2020-08-30T19:08:20.754 INFO:scrape: 2020-08-30T19:08:20.756 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [5383194] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench} ----------------------------------------------------------------- time: 03:16:48 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383194/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383194/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=26680fbfd34e41dfb5b2b11802db6919 Command failed (workunit test suites/blogbench.sh) on smithi105 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c0087509fbc6d51d305426be93ad7c5b765f9c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' [5383204] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:03:07 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383204/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383204/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=5cf8a32d4caa4de491dab067afb978d8 Command failed on smithi162 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl- devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs- devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' [5383208] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} ----------------------------------------------------------------- time: 00:58:16 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383208/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383208/ reached maximum tries (50) after waiting for 300 seconds Pass ================================================================= [5383195] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench} time: 00:51:55 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383195/ [5383196] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:12:02 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383196/ [5383197] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:12:58 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383197/ [5383198] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb} time: 01:12:38 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383198/ [5383199] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} time: 00:16:18 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383199/ [5383200] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 00:52:46 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383200/ [5383201] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:18:56 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383201/ [5383202] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx} time: 00:27:04 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383202/ [5383203] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd} time: 00:37:39 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383203/ [5383205] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383205/ [5383206] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} time: 00:36:26 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383206/ [5383207] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots} time: 00:24:58 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383207/ [5383209] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:19:10 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383209/ [5383210] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build} time: 00:52:36 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383210/ [5383211] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc} time: 01:07:07 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383211/ [5383212] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:13:05 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383212/ [5383213] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:09:43 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383213/ [5383214] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383214/ [5383215] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench} time: 00:32:39 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383215/ [5383216] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 00:57:03 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383216/ [5383217] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2020-08-28_13:15:04-multimds-octopus-distro-basic-smithi/5383217/