Found 306 jobs Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-14_21:00:02-rados-squid-distro-default-smithi/7601282/teuthology.log Found 11 distinct failure reasons Failure: "2024-03-22T00:30:00.000173+0000 mon.a (mon.0) 3108 : cluster 3 [WRN] TELEMETRY_CHANGED: Telemetry requires re-opt-in" in cluster log 16 jobs: ['7601439', '7601253', '7601267', '7601457', '7601286', '7601207', '7601481', '7601329', '7601314', '7601396', '7601230', '7601490', '7601415', '7601433', '7601296', '7601350'] suites intersection: [] suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Dead: None 6 jobs: ['7601282', '7601455', '7601451', '7601375', '7601454', '7601456'] Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6196 (apt-get)'] 1 jobs: ['7601471'] suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 4 jobs: ['7601239', '7601445', '7601385', '7601377'] suites intersection: [] suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] Dead: b'2024-03-22T08:29:12.474 DEBUG:teuthology.exit:Finished running handlers' 3 jobs: ['7601348', '7601284', '7601437'] suites intersection: [] suites union: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/cephadm/smoke/{0-distro/centos_9.stream_runc', 'rados/singleton-nomsgr/{all/admin_socket_output', 'start}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: SSH connection to smithi152 was lost: 'sudo yum install -y kernel' 1 jobs: ['7601460'] suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{centos_latest}}'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2 jobs: ['7601373', '7601219'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['7601308'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] Failure: Command failed on smithi080 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend' 2 jobs: ['7601408', '7601360'] suites intersection: ['mon_election/classic'] suites union: ['agent/off', 'clusters/9-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'task/test_ca_signed_key}', 'thrashers/one', 'workloads/rados_mon_workunits}'] Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5ff468cfa68f4f12d4cdfe7256729f6d89125f6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 1 jobs: ['7601252'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi175 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5ff468cfa68f4f12d4cdfe7256729f6d89125f6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 1 jobs: ['7601393'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_timeout}']