2023-03-21T22:48:22.117 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa
2023-03-21T22:48:22.118 INFO:teuthology.results:subset: '111/120000'
2023-03-21T22:48:22.118 INFO:teuthology.results:seed: '3360'
2023-03-22T09:47:51.381 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa
2023-03-22T09:47:51.412 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-22T12:38:05.220 INFO:teuthology.results:Tests finished! gathering results...
2023-03-22T12:38:10.886 INFO:scrape:Found 69 jobs
2023-03-22T12:38:15.236 INFO:scrape:Found 5 distinct failure reasons
2023-03-22T12:38:15.237 INFO:scrape:Failure: "2023-03-22T10:21:40.243609+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2023-03-22T12:38:15.238 INFO:scrape:3 jobs: ['7211755', '7211742', '7211692']
2023-03-22T12:38:15.238 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health']
2023-03-22T12:38:15.238 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/volumes/{overrides', 'test/clone}}']
2023-03-22T12:38:15.239 INFO:scrape:
2023-03-22T12:38:15.239 INFO:scrape:Failure: Command failed on smithi043 with status 1: 'set -ex\nsudo dd of=/home/ubuntu/cephtest/valgrind.supp'
2023-03-22T12:38:15.239 INFO:scrape:1 jobs: ['7211710']
2023-03-22T12:38:15.240 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_pjd}']
2023-03-22T12:38:15.240 INFO:scrape:
2023-03-22T12:38:15.240 INFO:scrape:Failure: "2023-03-22T00:12:26.223211+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 226833.495141 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-03-22T12:38:15.240 INFO:scrape:1 jobs: ['7211719']
2023-03-22T12:38:15.241 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}']
2023-03-22T12:38:15.241 INFO:scrape:
2023-03-22T12:38:15.242 INFO:scrape:Dead: b'2023-03-22T11:57:45.938 DEBUG:teuthology.exit:Finished running handlers'
2023-03-22T12:38:15.242 INFO:scrape:2 jobs: ['7211711', '7211726']
2023-03-22T12:38:15.242 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{frag', 'ranks/5']
2023-03-22T12:38:15.243 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'scrub/no', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'workunit/suites/fsstress}', 'wsync/{yes}}']
2023-03-22T12:38:15.243 INFO:scrape:
2023-03-22T12:38:15.243 INFO:scrape:Crash: Test failure: test_parallel_execution (tasks.cephfs.test_data_scan.TestDataScan)
2023-03-22T12:38:15.244 INFO:scrape:ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4cc090bb20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4cc190e4dd]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7f4cc190e649]
 6: (MDLog::_recovery_thread(MDSContext*)+0xb56) [0x55ba8b5e2096]
 7: (MDLog::RecoveryThread::entry()+0x15) [0x55ba8b296695]
 8: /lib64/libpthread.so.0(+0x814a) [0x7f4cc090114a]
 9: clone()
2023-03-22T12:38:15.244 INFO:scrape:1 jobs: ['7211707']
2023-03-22T12:38:15.244 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}']
2023-03-22T12:38:15.245 INFO:scrape: