Found 328 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2024-04-02_15:39:50-rados-wip-yuri2-testing-2024-04-01-1235-quincy-distro-default-smithi/7636570/teuthology.log Found 6 distinct failure reasons Dead: None 2 jobs: ['7636570', '7636571'] Failure: Command failed on smithi016 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49e8f32b3804bf60194367ce55a8ccc716a4fbdf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e91fa6b4-f164-11ee-b647-cb9ed24678a4 -- bash -c \'stat -c \'"\'"\'%u %g\'"\'"\' /var/log/ceph | grep \'"\'"\'167 167\'"\'"\'\'' 1 jobs: ['7636710'] suites: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'start}'] Dead: b'2024-04-03T14:19:55.525 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7636676'] suites: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] Failure: "1712110452.473959 mon.a (mon.0) 65 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 3 jobs: ['7636638', '7636449', '7636609'] suites intersection: [] suites union: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/objectstore/{backends/filestore-idempotent', 'rados/objectstore/{backends/filestore-idempotent-aio-journal', 'rados/singleton/{all/mon-config-keys', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: "1712110972.9349895 osd.11 (osd.11) 92 : cluster [ERR] 3.8s0 scrub : stat mismatch, got 30/30 objects, 24/24 clones, 30/30 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 1/1 whiteouts, 53149696/53395456 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 1 jobs: ['7636628'] suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] Failure: Exiting scrub checking -- not all pgs scrubbed. 1 jobs: ['7636646'] suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/osd_stale_reads', 'supported-random-distro$/{ubuntu_latest}}']