2021-01-24T04:15:00.312 INFO:root:teuthology version: 1.0.0-fe79ba1b 2021-01-24T04:15:00.313 INFO:teuthology.results:subset: '24/9999' 2021-01-24T04:15:00.313 INFO:teuthology.results:seed: '8642' 2021-01-24T04:33:05.852 INFO:root:teuthology version: 1.0.0-fe79ba1b 2021-01-24T04:33:05.873 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-24T08:10:09.527 INFO:teuthology.results:Tests finished! gathering results... 2021-01-24T08:10:14.281 INFO:scrape:Found 24 jobs 2021-01-24T08:10:20.424 INFO:scrape:Found 2 distinct failure reasons 2021-01-24T08:10:20.425 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/norstats/kernel_untar_tar.sh 2021-01-24T08:10:20.425 INFO:scrape:1 jobs: ['5804900'] 2021-01-24T08:10:20.425 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_latest}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_norstats}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-01-24T08:10:20.426 INFO:scrape: 2021-01-24T08:10:20.426 INFO:scrape:Failure: unexpected end of data 2021-01-24T08:10:20.427 INFO:scrape:1 jobs: ['5804891'] 2021-01-24T08:10:20.427 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/lockdep}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}'] 2021-01-24T08:10:20.428 INFO:scrape: 2021-01-24T08:10:20.430 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [5804891] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} ----------------------------------------------------------------- time: 01:23:55 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804891/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804891/ unexpected end of data [5804900] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats} ----------------------------------------------------------------- time: 03:35:29 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804900/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3311a157a8d349fe930650101223ad7d Command failed (workunit test fs/norstats/kernel_untar_tar.sh) on smithi081 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c35fa464478cb1593a8088402768df91f4a8066 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/norstats/kernel_untar_tar.sh' Pass ================================================================= [5804892] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} time: 00:39:52 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804892/ [5804893] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots} time: 00:23:55 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804893/ [5804894] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:36:12 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804894/ [5804895] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} time: 03:18:06 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804895/ [5804896] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build} time: 02:34:32 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804896/ [5804897] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc} time: 01:00:24 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804897/ [5804898] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:39:57 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804898/ [5804899] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} time: 01:23:09 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804899/ [5804901] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench} time: 01:05:34 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804901/ [5804902] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:39:43 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804902/ [5804903] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} time: 02:47:57 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804903/ [5804904] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench} time: 01:19:13 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804904/ [5804905] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb} time: 01:55:30 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804905/ [5804906] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:41:17 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804906/ [5804907] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:43:10 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804907/ [5804908] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} time: 00:39:33 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804908/ [5804909] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx} time: 00:49:46 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804909/ [5804910] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:20:21 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804910/ [5804911] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 01:00:32 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804911/ [5804912] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd} time: 00:35:19 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804912/ [5804913] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} time: 00:42:27 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804913/ [5804914] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:42:21 info: http://pulpito.ceph.com/teuthology-2021-01-20_13:15:03-multimds-octopus-distro-basic-smithi/5804914/