2021-12-17T08:44:40.474 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-17T08:44:40.475 INFO:teuthology.results:subset: '25/32'
2021-12-17T08:44:40.476 INFO:teuthology.results:seed: '3715'
2021-12-17T08:48:14.236 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-17T08:48:14.242 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-12-17T11:16:25.498 INFO:teuthology.results:Tests finished! gathering results...
2021-12-17T11:16:26.272 INFO:scrape:Found 9 jobs
2021-12-17T11:16:29.357 INFO:scrape:Found 4 distinct failure reasons
2021-12-17T11:16:29.358 INFO:scrape:Failure: Command failed on smithi122 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01dccc3a-5f19-11ec-8c31-001a4aab830c -- bash -c \'ceph --format=json mds versions | jq -e ". | add == 4"\''
2021-12-17T11:16:29.358 INFO:scrape:2 jobs: ['6568034', '6568030']
2021-12-17T11:16:29.358 INFO:scrape:suites intersection: []
2021-12-17T11:16:29.359 INFO:scrape:suites union: ['1-mount', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-workload/dbench', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'distro/centos_latest}', 'fs/cephadm/multivolume/{0-start', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-17T11:16:29.359 INFO:scrape:
2021-12-17T11:16:29.360 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2021-12-17T11:16:29.360 INFO:scrape:1 jobs: ['6568029']
2021-12-17T11:16:29.361 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-17T11:16:29.361 INFO:scrape:
2021-12-17T11:16:29.362 INFO:scrape:Failure: "2021-12-17T09:19:55.273669+0000 mds.g (mds.1) 33 : cluster [WRN] Scrub error on inode 0x10000003f43 (/client.0/tmp/t/linux-5.4/drivers/dca/Kconfig) see mds.g log and `damage ls` output for details" in cluster log
2021-12-17T11:16:29.362 INFO:scrape:1 jobs: ['6568032']
2021-12-17T11:16:29.362 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/kernel_untar_build}', 'wsync/{yes}}']
2021-12-17T11:16:29.363 INFO:scrape:
2021-12-17T11:16:29.363 INFO:scrape:Crash: Command failed (workunit test kernel_untar_build.sh) on smithi043 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c9f46754c17c9bc381091a4ebba4aff4ddaf306f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'
2021-12-17T11:16:29.364 INFO:scrape:ceph version 17.0.0-9669-gc9f46754 (c9f46754c17c9bc381091a4ebba4aff4ddaf306f) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f13ce8d0c20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f13d093bd12]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x281ed5) [0x7f13d093bed5]
 6: (Monitor::~Monitor()+0xb35) [0x55d515f19625]
 7: (Monitor::~Monitor()+0xd) [0x55d515f1967d]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-12-17T11:16:29.364 INFO:scrape:1 jobs: ['6568036']
2021-12-17T11:16:29.365 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8.stream}', 'fs/workload/{begin', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/kernel_untar_build}', 'wsync/{no}}']
2021-12-17T11:16:29.365 INFO:scrape:
2021-12-17T11:16:29.366 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  4


Fail
=================================================================
[6568029]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
-----------------------------------------------------------------
time:   00:13:42
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568029/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568029/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8af8611a8cc24b308d6de7cefb646dd1

    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)

[6568030]  fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
-----------------------------------------------------------------
time:   00:11:05
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568030/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568030/
sentry: https://sentry.ceph.com/organizations/ceph/?query=66de4773319643198a8650211ae4cda2

    Command failed on smithi064 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:c9f46754c17c9bc381091a4ebba4aff4ddaf306f pull'

[6568032]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
-----------------------------------------------------------------
time:   00:46:40
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568032/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568032/

    "2021-12-17T09:19:55.273669+0000 mds.g (mds.1) 33 : cluster [WRN] Scrub
    error on inode 0x10000003f43
    (/client.0/tmp/t/linux-5.4/drivers/dca/Kconfig) see mds.g log and `damage
    ls` output for details" in cluster log

[6568034]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:16:25
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568034/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568034/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9d5c7a184ab34e5090a33909bba75f84

    Command failed on smithi122 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    01dccc3a-5f19-11ec-8c31-001a4aab830c -- bash -c \'ceph --format=json mds
    versions | jq -e ". | add == 4"\''

[6568036]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
-----------------------------------------------------------------
time:   00:38:50
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568036/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568036/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1756365c75a743d797e5883b9e88cfe2

    Command failed (workunit test kernel_untar_build.sh) on smithi043 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c9f46754c17c9bc381091a4ebba4aff4ddaf306f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'



Pass
=================================================================
[6568031] fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:19:38
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568031/

[6568033] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:36:43
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568033/

[6568035] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:43:05
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568035/

[6568037] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:02:39
info:   http://pulpito.ceph.com/vshankar-2021-12-17_08:44:26-fs-wip-vshankar-testing-20211216-114012-testing-default-smithi/6568037/