2022-08-30T16:02:17.111 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-08-30T16:02:17.114 INFO:teuthology.results:subset: '111/120000'
2022-08-30T16:02:17.114 INFO:teuthology.results:seed: '6228'
2022-08-30T18:01:03.649 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-08-30T18:01:03.679 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-08-30T19:26:18.444 INFO:teuthology.results:Tests finished! gathering results...
2022-08-30T19:26:30.604 INFO:scrape:Found 114 jobs
2022-08-30T19:26:33.245 INFO:scrape:Found 5 distinct failure reasons
2022-08-30T19:26:33.247 INFO:scrape:Failure: Command failed on smithi079 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'
2022-08-30T19:26:33.247 INFO:scrape:6 jobs: ['7000591', '7000593', '7000594', '7000588', '7000592', '7000587']
2022-08-30T19:26:33.248 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}']
2022-08-30T19:26:33.249 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_suites_pjd}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/multifs/{begin', 'fs/snaps/{begin', 'fs/thrash/multifs/{begin', 'fs/traceless/{begin', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{whitelist_health', 'rhel_8}', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/multifs-auth}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'thrashosds-health', 'traceless/50pc}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-08-30T19:26:33.249 INFO:scrape:
2022-08-30T19:26:33.250 INFO:scrape:Failure: "2022-08-30T16:45:38.208075+0000 mon.a (mon.0) 351 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log
2022-08-30T19:26:33.251 INFO:scrape:1 jobs: ['7000604']
2022-08-30T19:26:33.251 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-08-30T19:26:33.252 INFO:scrape:
2022-08-30T19:26:33.252 INFO:scrape:Failure: Command failed on smithi105 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ca0f5862-2881-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi105:vg_nvme/lv_4'
2022-08-30T19:26:33.253 INFO:scrape:1 jobs: ['7000627']
2022-08-30T19:26:33.253 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-08-30T19:26:33.254 INFO:scrape:
2022-08-30T19:26:33.254 INFO:scrape:Failure: Command failed on smithi099 with status 1: 'sudo yum -y install ceph-test'
2022-08-30T19:26:33.255 INFO:scrape:1 jobs: ['7000590']
2022-08-30T19:26:33.255 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-08-30T19:26:33.255 INFO:scrape:
2022-08-30T19:26:33.256 INFO:scrape:Failure: "2022-08-30T16:57:26.788453+0000 mds.a (mds.2) 2 : cluster [WRN] client.4986 isn't responding to mclientcaps(revoke), ino 0x10000002236 pending pAsLsXs issued pAsLsXsFsx, sent 300.002734 seconds ago" in cluster log
2022-08-30T19:26:33.256 INFO:scrape:1 jobs: ['7000625']
2022-08-30T19:26:33.257 INFO:scrape:suites: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-08-30T19:26:33.257 INFO:scrape: