2024-04-11T18:03:22.392 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-11T18:03:22.392 INFO:teuthology.results:subset: '210/512'
2024-04-11T18:03:22.393 INFO:teuthology.results:seed: '7925'
2024-04-11T18:17:37.364 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-11T18:17:37.367 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-11T20:24:47.240 INFO:teuthology.results:Tests finished! gathering results...
2024-04-11T20:24:48.441 INFO:scrape:Found 14 jobs
2024-04-11T20:24:53.718 INFO:scrape:Found 6 distinct failure reasons
2024-04-11T20:24:53.719 INFO:scrape:Failure: "1712860967.1498935 mon.a (mon.0) 1171 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log
2024-04-11T20:24:53.719 INFO:scrape:1 jobs: ['7652534']
2024-04-11T20:24:53.719 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}']
2024-04-11T20:24:53.719 INFO:scrape:
2024-04-11T20:24:53.719 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
2024-04-11T20:24:53.719 INFO:scrape:2 jobs: ['7652542', '7652540']
2024-04-11T20:24:53.719 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse']
2024-04-11T20:24:53.719 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}', 'tasks/mirror}}']
2024-04-11T20:24:53.720 INFO:scrape:
2024-04-11T20:24:53.720 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi179 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=eb10a30c5f65b6627c442360f63166db225304ae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2024-04-11T20:24:53.720 INFO:scrape:2 jobs: ['7652545', '7652538']
2024-04-11T20:24:53.720 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2024-04-11T20:24:53.720 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2024-04-11T20:24:53.720 INFO:scrape:
2024-04-11T20:24:53.720 INFO:scrape:Failure: "1712860816.4778037 osd.1 (osd.1) 3 : cluster [WRN] OSD bench result of 135819.032910 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.1. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2024-04-11T20:24:53.720 INFO:scrape:1 jobs: ['7652544']
2024-04-11T20:24:53.720 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/mds-full}']
2024-04-11T20:24:53.721 INFO:scrape:
2024-04-11T20:24:53.721 INFO:scrape:Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS)
2024-04-11T20:24:53.721 INFO:scrape:1 jobs: ['7652541']
2024-04-11T20:24:53.721 INFO:scrape:suites: ['fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{rhel_8}', 'tasks/nfs}']
2024-04-11T20:24:53.721 INFO:scrape:
2024-04-11T20:24:53.721 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=eb10a30c5f65b6627c442360f63166db225304ae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2024-04-11T20:24:53.721 INFO:scrape:1 jobs: ['7652543']
2024-04-11T20:24:53.721 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', '3-snaps/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/ffsb}', 'wsync/{no}}']
2024-04-11T20:24:53.721 INFO:scrape: