2022-02-24T22:37:02.865 INFO:root:teuthology version: 1.1.0-5825e8d0
2022-02-24T22:37:02.867 INFO:teuthology.results:subset: '111/120000'
2022-02-24T22:37:02.867 INFO:teuthology.results:seed: '2245'
2022-02-24T23:25:48.839 INFO:root:teuthology version: 1.1.0-5825e8d0
2022-02-24T23:25:48.895 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-02-25T01:11:57.155 INFO:teuthology.results:Tests finished! gathering results...
2022-02-25T01:12:06.241 INFO:scrape:Found 65 jobs
2022-02-25T01:12:13.467 WARNING:scrape:Missing log /home/teuthworker/archive/yuriw-2022-02-24_22:32:08-fs-wip-yuri3-testing-2022-02-22-0946-pacific-distro-default-smithi/6704781/remote/smithi186/log/ceph-mgr.z.log.gz
2022-02-25T01:12:13.468 WARNING:scrape:Not-understood crash indication 
2022-02-25T01:12:13.508 WARNING:scrape:Missing log /home/teuthworker/archive/yuriw-2022-02-24_22:32:08-fs-wip-yuri3-testing-2022-02-22-0946-pacific-distro-default-smithi/6704781/remote/smithi186/log/ceph-mgr.z.log.gz
2022-02-25T01:12:13.510 WARNING:scrape:Not-understood crash indication 
2022-02-25T01:12:13.557 WARNING:scrape:Missing log /home/teuthworker/archive/yuriw-2022-02-24_22:32:08-fs-wip-yuri3-testing-2022-02-22-0946-pacific-distro-default-smithi/6704781/remote/smithi186/log/ceph-mgr.z.log.gz
2022-02-25T01:12:13.558 WARNING:scrape:Not-understood crash indication 
2022-02-25T01:12:13.601 WARNING:scrape:Missing log /home/teuthworker/archive/yuriw-2022-02-24_22:32:08-fs-wip-yuri3-testing-2022-02-22-0946-pacific-distro-default-smithi/6704781/remote/smithi186/log/ceph-mgr.z.log.gz
2022-02-25T01:12:13.602 WARNING:scrape:Not-understood crash indication 
2022-02-25T01:12:14.157 INFO:scrape:Found 8 distinct failure reasons
2022-02-25T01:12:14.159 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2022-02-25T01:12:14.159 INFO:scrape:1 jobs: ['6704827']
2022-02-25T01:12:14.160 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.163 INFO:scrape:
2022-02-25T01:12:14.164 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2022-02-25T01:12:14.164 INFO:scrape:1 jobs: ['6704808']
2022-02-25T01:12:14.165 INFO:scrape:suites: ['centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-02-25T01:12:14.166 INFO:scrape:
2022-02-25T01:12:14.167 INFO:scrape:Failure: "2022-02-24T23:15:54.757482+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2022-02-25T01:12:14.167 INFO:scrape:1 jobs: ['6704802']
2022-02-25T01:12:14.168 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.168 INFO:scrape:
2022-02-25T01:12:14.169 INFO:scrape:Failure: Command failed on smithi154 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2022-02-25T01:12:14.169 INFO:scrape:4 jobs: ['6704810', '6704791', '6704842', '6704781']
2022-02-25T01:12:14.170 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.170 INFO:scrape:suites union: ['2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'fs/verify/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'overrides/{mon-debug', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/fsstress', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'thrashosds-health', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}', 'wsync/{no}}']
2022-02-25T01:12:14.171 INFO:scrape:
2022-02-25T01:12:14.171 INFO:scrape:Failure: Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc)
2022-02-25T01:12:14.172 INFO:scrape:2 jobs: ['6704785', '6704836']
2022-02-25T01:12:14.172 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.173 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.173 INFO:scrape:
2022-02-25T01:12:14.173 INFO:scrape:Failure: Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays)
2022-02-25T01:12:14.174 INFO:scrape:1 jobs: ['6704803']
2022-02-25T01:12:14.174 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.175 INFO:scrape:
2022-02-25T01:12:14.175 INFO:scrape:Failure: Command failed on smithi181 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 497ab548-95c9-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi181:vg_nvme/lv_4'
2022-02-25T01:12:14.176 INFO:scrape:1 jobs: ['6704821']
2022-02-25T01:12:14.176 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.177 INFO:scrape:
2022-02-25T01:12:14.177 INFO:scrape:Failure: Command failed on smithi195 with status 1: "sudo nsenter --net=/var/run/netns/ceph-ns-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' --id vol_data_isolated --client_mountpoint=/volumes/_nogroup/vol_isolated mnt.0"
2022-02-25T01:12:14.177 INFO:scrape:1 jobs: ['6704818']
2022-02-25T01:12:14.178 INFO:scrape:suites: ['1-client', '2-upgrade', '3-verify}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/volumes/import-legacy/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'tasks/{0-nautilus', 'ubuntu_18.04}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-25T01:12:14.178 INFO:scrape: