2022-04-28T20:18:05.199 INFO:root:teuthology version: 1.1.1.dev328+g004701de
2022-04-28T20:18:05.199 INFO:teuthology.results:subset: '12/32'
2022-04-28T20:18:05.199 INFO:teuthology.results:seed: '2665'
2022-04-28T20:33:41.013 INFO:root:teuthology version: 1.1.1.dev328+g004701de
2022-04-28T20:33:41.019 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-04-28T23:56:56.515 INFO:teuthology.results:Tests finished! gathering results...
2022-04-28T23:56:57.197 INFO:scrape:Found 8 jobs
2022-04-28T23:57:00.286 INFO:scrape:Found 5 distinct failure reasons
2022-04-28T23:57:00.286 INFO:scrape:Failure: Command failed on smithi122 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)"
2022-04-28T23:57:00.286 INFO:scrape:3 jobs: ['6811842', '6811846', '6811844']
2022-04-28T23:57:00.287 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/none', 'osd}', 'overrides/{frag', 'prefetch_entire_dirfrags/no', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-28T23:57:00.287 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-28T23:57:00.287 INFO:scrape:
2022-04-28T23:57:00.287 INFO:scrape:Failure: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0581ef2c21896c4f42abeb945bb9ce30e0389903 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'
2022-04-28T23:57:00.287 INFO:scrape:1 jobs: ['6811848']
2022-04-28T23:57:00.288 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-28T23:57:00.288 INFO:scrape:
2022-04-28T23:57:00.288 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0581ef2c21896c4f42abeb945bb9ce30e0389903 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2022-04-28T23:57:00.288 INFO:scrape:1 jobs: ['6811841']
2022-04-28T23:57:00.288 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:thrash/multifs/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-28T23:57:00.288 INFO:scrape:
2022-04-28T23:57:00.288 INFO:scrape:Failure: Timed out waiting for MDS daemons to become healthy
2022-04-28T23:57:00.289 INFO:scrape:1 jobs: ['6811847']
2022-04-28T23:57:00.289 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-28T23:57:00.289 INFO:scrape:
2022-04-28T23:57:00.289 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh
2022-04-28T23:57:00.289 INFO:scrape:1 jobs: ['6811843']
2022-04-28T23:57:00.289 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-28T23:57:00.289 INFO:scrape:
2022-04-28T23:57:00.290 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi
=================================================================
info:    http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/
failed:  7
dead:    0
running: 0
waiting: 0
queued:  0
passed:  1


Fail
=================================================================
[6811841]  fs:thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:44:56
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811841/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811841/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7cc36dab2c9f49888dacbf7a5f79d5ae

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi007 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0581ef2c21896c4f42abeb945bb9ce30e0389903
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[6811842]  fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:10:01
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811842/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811842/
sentry: https://sentry.ceph.com/organizations/ceph/?query=74312d58e3764e76b0a918dec6c71a5c

    Command failed on smithi122 with status 1: "(cd /home/ubuntu/cephtest &&
    exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-
    mnt.admin sudo adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f
    --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok'
    /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)"

[6811843]  fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   03:25:21
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811843/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c97ae02229054e9dbfe423c1700e1372

    Command failed (workunit test suites/ffsb.sh) on smithi137 with status 124:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0581ef2c21896c4f42abeb945bb9ce30e0389903
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[6811844]  fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:11:47
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811844/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811844/
sentry: https://sentry.ceph.com/organizations/ceph/?query=65a0c262562048ec9ace52fccd86f1ab

    Command failed on smithi079 with status 1: "(cd /home/ubuntu/cephtest &&
    exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-
    mnt.admin sudo adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f
    --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok'
    /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)"

[6811846]  fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:14:56
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811846/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811846/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7ae04b0c73a84dad97964624643f43f3

    Command failed on smithi008 with status 1: "(cd /home/ubuntu/cephtest &&
    exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-
    mnt.admin sudo adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f
    --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok'
    /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)"

[6811847]  fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:26:57
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811847/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811847/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6cf1b9649ddb49dfb3d542ac7bbfea94

    Timed out waiting for MDS daemons to become healthy

[6811848]  fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:20:59
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811848/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811848/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cf5bffa50dfc4a5da38497713d66d888

    Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi078 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=0581ef2c21896c4f42abeb945bb9ce30e0389903
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'



Pass
=================================================================
[6811845] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:12:42
info:   http://pulpito.ceph.com/jlayton-2022-04-28_20:14:31-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6811845/