2023-08-09T20:01:01.099 INFO:root:teuthology version: 0.0.1.dev181+g7fda959
2023-08-09T20:01:01.137 INFO:teuthology.results:subset: '111/120000'
2023-08-09T20:01:01.137 INFO:teuthology.results:seed: '3482'
2023-08-09T20:44:37.627 INFO:root:teuthology version: 0.0.1.dev181+g7fda959
2023-08-09T20:44:37.647 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-09T23:03:48.321 INFO:teuthology.results:Tests finished! gathering results...
2023-08-09T23:03:49.422 INFO:scrape:Found 13 jobs
2023-08-09T23:03:53.906 INFO:scrape:Found 6 distinct failure reasons
2023-08-09T23:03:53.907 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2023-08-09T23:03:53.907 INFO:scrape:1 jobs: ['7364389']
2023-08-09T23:03:53.908 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/acls}']
2023-08-09T23:03:53.909 INFO:scrape:
2023-08-09T23:03:53.909 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c99915ec04212fb433eef93cd14cd65cd72d46b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-08-09T23:03:53.910 INFO:scrape:1 jobs: ['7364390']
2023-08-09T23:03:53.910 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-08-09T23:03:53.910 INFO:scrape:
2023-08-09T23:03:53.911 INFO:scrape:Failure: "2023-08-09T21:04:44.497188+0000 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 223693.062100 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-08-09T23:03:53.911 INFO:scrape:1 jobs: ['7364395']
2023-08-09T23:03:53.912 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}']
2023-08-09T23:03:53.912 INFO:scrape:
2023-08-09T23:03:53.912 INFO:scrape:Failure: "2023-08-09T21:06:01.909115+0000 mon.a (mon.0) 377 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2023-08-09T23:03:53.913 INFO:scrape:1 jobs: ['7364387']
2023-08-09T23:03:53.913 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2023-08-09T23:03:53.913 INFO:scrape:
2023-08-09T23:03:53.913 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c99915ec04212fb433eef93cd14cd65cd72d46b4
2023-08-09T23:03:53.914 INFO:scrape:2 jobs: ['7364394', '7364386']
2023-08-09T23:03:53.914 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-08-09T23:03:53.914 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-08-09T23:03:53.915 INFO:scrape:
2023-08-09T23:03:53.915 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2023-08-09T23:03:53.915 INFO:scrape:1 jobs: ['7364391']
2023-08-09T23:03:53.916 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{rhel_8}', 'workloads/cephfs-mirror-ha-workunit}']
2023-08-09T23:03:53.917 INFO:scrape: