2021-04-03T13:15:40.681 INFO:root:teuthology version: 1.1.0-85d61eae
2021-04-03T13:15:40.684 INFO:teuthology.results:subset: '5936/9999'
2021-04-03T13:15:40.684 INFO:teuthology.results:seed: '1151'
2021-04-03T14:52:18.724 INFO:root:teuthology version: 1.1.0-85d61eae
2021-04-03T14:52:18.756 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-04-04T02:28:54.041 INFO:teuthology.results:Tests finished! gathering results...
2021-04-04T02:28:56.713 INFO:scrape:Found 24 jobs
2021-04-04T02:28:57.587 INFO:scrape:Found 3 distinct failure reasons
2021-04-04T02:28:57.588 INFO:scrape:Failure: "2021-04-03T15:36:51.718673+0000 mds.f (mds.2) 1 : cluster [WRN] client.44510 isn't responding to mclientcaps(revoke), ino 0x300000008c9 pending pAsLsXsFr issued pAsLsXsFscr, sent 303.799645 seconds ago" in cluster log
2021-04-04T02:28:57.588 INFO:scrape:1 jobs: ['6017895']
2021-04-04T02:28:57.589 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-04-04T02:28:57.589 INFO:scrape:
2021-04-04T02:28:57.590 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on gibba042 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5feb67b5bda898b8c739180363606432d4f39831 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'
2021-04-04T02:28:57.590 INFO:scrape:1 jobs: ['6017892']
2021-04-04T02:28:57.591 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_suites_fsx}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-04-04T02:28:57.591 INFO:scrape:
2021-04-04T02:28:57.592 INFO:scrape:Dead: b'2021-04-04T02:25:52.212 DEBUG:teuthology.exit:Finished running handlers'
2021-04-04T02:28:57.592 INFO:scrape:1 jobs: ['6017887']
2021-04-04T02:28:57.593 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-04-04T02:28:57.593 INFO:scrape:
2021-04-04T02:28:57.595 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba
=================================================================
info:    http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/
failed:  2
dead:    1
running: 0
waiting: 0
queued:  0
passed:  21


Fail
=================================================================
[6017892]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
-----------------------------------------------------------------
time:   00:12:35
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017892/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017892/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b35bb9d17d4e4bb99342fecd16953d31

    Command failed (workunit test suites/fsx.sh) on gibba042 with status 2:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5feb67b5bda898b8c739180363606432d4f39831
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

[6017895]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:50:11
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017895/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017895/

    "2021-04-03T15:36:51.718673+0000 mds.f (mds.2) 1 : cluster [WRN]
    client.44510 isn't responding to mclientcaps(revoke), ino 0x300000008c9
    pending pAsLsXsFr issued pAsLsXsFscr, sent 303.799645 seconds ago" in
    cluster log



Dead
=================================================================
[6017887]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017887/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017887/

    hit max job timeout



Pass
=================================================================
[6017874] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:12:22
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017874/

[6017875] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:15:21
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017875/

[6017876] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:24:09
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017876/

[6017877] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:26:49
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017877/

[6017878] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:11:56
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017878/

[6017879] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   01:18:22
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017879/

[6017880] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:37:44
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017880/

[6017881] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:11:11
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017881/

[6017882] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:11:09
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017882/

[6017883] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:11:25
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017883/

[6017884] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:33:32
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017884/

[6017885] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:52:08
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017885/

[6017886] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:11:39
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017886/

[6017888] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:32:12
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017888/

[6017889] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:13:42
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017889/

[6017890] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:12:43
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017890/

[6017891] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:12:08
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017891/

[6017893] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:12:01
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017893/

[6017894] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:10:45
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017894/

[6017896] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:36:44
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017896/

[6017897] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:25:55
info:   http://pulpito.ceph.com/teuthology-2021-04-03_13:15:03-multimds-octopus-distro-basic-gibba/6017897/