Found 14 jobs Found 8 distinct failure reasons Failure: Test failure: test_cephfs_mirror_cancel_sync (tasks.cephfs.test_mirroring.TestMirroring) 1 jobs: ['7652329'] suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) 1 jobs: ['7652324'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] Failure: Test failure: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules), test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 1 jobs: ['7652321'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}'] Failure: Command failed (workunit test kernel_untar_build.sh) on smithi165 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ef705e8073d287af41c39814aead7fb68d4004e8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2 jobs: ['7652332', '7652325'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] Failure: Failed to reconnect to smithi179 2 jobs: ['7652322', '7652330'] suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'session_timeout}'] suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '3-snaps/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/fsstress', 'tasks/{0-check-counter', 'validater/valgrind}', 'workunit/suites/ffsb}', 'wsync/{no}}'] Failure: Test failure: test_cephfs_mirror_incremental_sync (tasks.cephfs.test_mirroring.TestMirroring), test_cephfs_mirror_incremental_sync (tasks.cephfs.test_mirroring.TestMirroring) 1 jobs: ['7652327'] suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror/{begin/{0-install', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/{ignorelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}'] Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) 1 jobs: ['7652328'] suites: ['fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{rhel_8}', 'tasks/nfs}'] Failure: "1712854548.3365893 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 135764.870089 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 1 jobs: ['7652331'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/mds-full}']