Found 195 jobs
Missing teuthology log /home/teuthworker/archive/vshankar-2024-03-14_16:52:41-fs-wip-vshankar-testing1-quincy-2024-03-14-0655-quincy-testing-default-smithi/7600954/teuthology.log
Found 15 distinct failure reasons
Dead: None
5 jobs: ['7600954', '7601021', '7600955', '7600971', '7600953']

Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0349feef6e1f7e72fa54de053175c84ee6c2d8a0
2 jobs: ['7600884', '7600989']
suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']

Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize)
1 jobs: ['7600939']
suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/admin}']

Failure: ["error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)"]
1 jobs: ['7600957']
suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}']

Failure: ['Data could not be sent to remote host "smithi112.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi112.front.sepia.ceph.com port 22: No route to host']
1 jobs: ['7600958']
suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_blogbench', 'traceless/50pc}']

Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps)
3 jobs: ['7600920', '7600983', '7600856']
suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}']

Dead: b'2024-03-18T04:26:31.991 DEBUG:teuthology.exit:Finished running handlers'
1 jobs: ['7600987']
suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}']

Failure: "1710681502.0206146 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 209186.757195 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2 jobs: ['7600847', '7601010']
suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}']
suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}']

Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
1 jobs: ['7600927']
suites: ['1-cephadm', '2-logrotate}', '3-snaps/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/5', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/iogen}', 'wsync/{yes}}']

Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
1 jobs: ['7600914']
suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}']

Failure: "1710687244.1943853 mon.a (mon.0) 830 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
3 jobs: ['7600900', '7601013', '7600868']
suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/failover}', 'tasks/metrics}', 'tasks/snap-schedule}']

Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
1 jobs: ['7600879']
suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']

Failure: "1710701528.6048803 osd.1 (osd.1) 6 : cluster [ERR] 4.2s0 deep-scrub : stat mismatch, got 0/1 objects, 0/0 clones, 0/1 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/931 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
1 jobs: ['7601019']
suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'prefetch_dirfrags/yes', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']

Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
2 jobs: ['7600853', '7600862']
suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse']
suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}', 'tasks/mirror}}']

Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS)
1 jobs: ['7600857']
suites: ['fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}']