2022-01-30T09:30:59.193 INFO:root:teuthology version: 1.1.0-7c0cb867
2022-01-30T09:30:59.194 INFO:teuthology.results:subset: '618/9999'
2022-01-30T09:30:59.195 INFO:teuthology.results:seed: '4720'
2022-01-30T10:59:36.225 INFO:root:teuthology version: 1.1.0-7c0cb867
2022-01-30T10:59:36.253 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-01-30T17:37:06.097 INFO:teuthology.results:Tests finished! gathering results...
2022-01-30T17:37:08.524 INFO:scrape:Found 24 jobs
2022-01-30T17:37:11.928 INFO:scrape:Found 4 distinct failure reasons
2022-01-30T17:37:11.929 INFO:scrape:Timeout 6h running clone.client.1/qa/workunits/suites/fsstress.sh
2022-01-30T17:37:11.930 INFO:scrape:1 jobs: ['6648217']
2022-01-30T17:37:11.930 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-01-30T17:37:11.931 INFO:scrape:
2022-01-30T17:37:11.931 INFO:scrape:Dead: b'2022-01-30T17:34:21.607 DEBUG:teuthology.exit:Finished running handlers'
2022-01-30T17:37:11.932 INFO:scrape:1 jobs: ['6648222']
2022-01-30T17:37:11.932 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2022-01-30T17:37:11.933 INFO:scrape:
2022-01-30T17:37:11.933 INFO:scrape:Failure: SSH connection to smithi146 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1'
2022-01-30T17:37:11.934 INFO:scrape:1 jobs: ['6648211']
2022-01-30T17:37:11.934 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-01-30T17:37:11.935 INFO:scrape:
2022-01-30T17:37:11.935 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
2022-01-30T17:37:11.936 INFO:scrape:1 jobs: ['6648201']
2022-01-30T17:37:11.936 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-01-30T17:37:11.937 INFO:scrape:
2022-01-30T17:37:11.938 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/
failed:  2
dead:    2
running: 0
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[6648201]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:42:52
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648201/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648201/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}

[6648217]  multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   06:18:31
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648217/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648217/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a4f036bdba8543d88f5a2ba21f418ed4

    Command failed (workunit test suites/fsstress.sh) on smithi154 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=9f723519257eca039126a20aa6a2a7d2dbfb5dba
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'



Dead
=================================================================
[6648211]  multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
-----------------------------------------------------------------
time:   00:37:33
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648211/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648211/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d823085869b14c73bf71079b5f56a5b0

    SSH connection to smithi146 was lost: 'sudo rm -rf --
    /home/ubuntu/cephtest/workunits.list.client.1
    /home/ubuntu/cephtest/clone.client.1'

[6648222]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648222/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648222/

    hit max job timeout



Pass
=================================================================
[6648199] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:13:38
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648199/

[6648200] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:22
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648200/

[6648202] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:56:40
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648202/

[6648203] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:24:59
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648203/

[6648204] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:49
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648204/

[6648205] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   02:02:56
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648205/

[6648206] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   02:16:52
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648206/

[6648207] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:50:48
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648207/

[6648208] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:54
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648208/

[6648209] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:20:11
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648209/

[6648210] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:56:35
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648210/

[6648212] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:41
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648212/

[6648213] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:56:32
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648213/

[6648214] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   01:58:52
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648214/

[6648215] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:30:17
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648215/

[6648216] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:53
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648216/

[6648218] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:15:30
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648218/

[6648219] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:22:06
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648219/

[6648220] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:52
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648220/

[6648221] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:10:59
info:   http://pulpito.ceph.com/teuthology-2022-01-29_13:15:03-multimds-octopus-distro-default-smithi/6648221/