2023-02-16T19:10:02.577 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5
2023-02-16T19:10:02.578 INFO:teuthology.results:subset: '111/120000'
2023-02-16T19:10:02.578 INFO:teuthology.results:seed: '131'
2023-02-16T19:44:14.153 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5
2023-02-16T19:44:14.157 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-02-17T07:43:08.369 INFO:teuthology.results:Tests finished! gathering results...
2023-02-17T07:43:11.373 INFO:scrape:Found 35 jobs
2023-02-17T07:43:23.561 INFO:scrape:Found 9 distinct failure reasons
2023-02-17T07:43:23.563 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fd249f18f047d5b86a29796a05bfeba7a7666d84
2023-02-17T07:43:23.563 INFO:scrape:2 jobs: ['7176554', '7176569']
2023-02-17T07:43:23.563 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-02-17T07:43:23.564 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-02-17T07:43:23.564 INFO:scrape:
2023-02-17T07:43:23.564 INFO:scrape:Failure: "2023-02-16T19:45:04.299293+0000 osd.7 (osd.7) 3 : cluster [WRN] OSD bench result of 223745.181088 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-02-17T07:43:23.564 INFO:scrape:1 jobs: ['7176557']
2023-02-17T07:43:23.565 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}']
2023-02-17T07:43:23.565 INFO:scrape:
2023-02-17T07:43:23.565 INFO:scrape:Failure: "2023-02-16T20:11:00.337224+0000 mds.c (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi035:0 (6617), after waiting 48.287 seconds during MDS startup" in cluster log
2023-02-17T07:43:23.566 INFO:scrape:1 jobs: ['7176568']
2023-02-17T07:43:23.566 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}']
2023-02-17T07:43:23.566 INFO:scrape:
2023-02-17T07:43:23.566 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2023-02-17T07:43:23.567 INFO:scrape:2 jobs: ['7176567', '7176542']
2023-02-17T07:43:23.567 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2023-02-17T07:43:23.567 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2023-02-17T07:43:23.568 INFO:scrape:
2023-02-17T07:43:23.568 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-02-17T07:43:23.568 INFO:scrape:1 jobs: ['7176561']
2023-02-17T07:43:23.569 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2023-02-17T07:43:23.569 INFO:scrape:
2023-02-17T07:43:23.569 INFO:scrape:Failure: "2023-02-16T20:08:10.489623+0000 mon.a (mon.0) 976 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2023-02-17T07:43:23.569 INFO:scrape:1 jobs: ['7176548']
2023-02-17T07:43:23.570 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2023-02-17T07:43:23.570 INFO:scrape:
2023-02-17T07:43:23.570 INFO:scrape:Dead: b'2023-02-17T07:36:20.056 DEBUG:teuthology.exit:Finished running handlers'
2023-02-17T07:43:23.571 INFO:scrape:1 jobs: ['7176556']
2023-02-17T07:43:23.571 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-02-17T07:43:23.571 INFO:scrape:
2023-02-17T07:43:23.572 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi073 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive'
2023-02-17T07:43:23.572 INFO:scrape:1 jobs: ['7176546']
2023-02-17T07:43:23.572 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/dbench}', 'wsync/{yes}}']
2023-02-17T07:43:23.572 INFO:scrape:
2023-02-17T07:43:23.573 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2023-02-17T07:43:23.573 INFO:scrape:1 jobs: ['7176576']
2023-02-17T07:43:23.573 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}']
2023-02-17T07:43:23.574 INFO:scrape: