Found 297 jobs
Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628793/teuthology.log
Found 13 distinct failure reasons
Failure: "2024-03-29T06:50:00.000128+0000 mon.a (mon.0) 3478 : cluster 3 [WRN] TELEMETRY_CHANGED: Telemetry requires re-opt-in" in cluster log
17 jobs: ['7628642', '7628625', '7628595', '7628691', '7628554', '7628671', '7628564', '7628614', '7628774', '7628648', '7628608', '7628787', '7628741', '7628707', '7628751', '7628829', '7628813']
suites intersection: []
suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'agent/on', 'mon_election/classic', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'task/test_extra_daemon_features}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']

Failure: Command failed (workunit test mon/osd-erasure-code-profile.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-erasure-code-profile.sh'
1 jobs: ['7628820']
suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/mon}']

Failure: Command failed (workunit test cephtool/test.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
4 jobs: ['7628592', '7628810', '7628667', '7628740']
suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}']

Failure: Command failed on smithi151 with status 5: 'sudo systemctl stop ceph-8d6739f0-ed94-11ee-b647-cb9ed24678a4@mon.a'
1 jobs: ['7628701']
suites: ['agent/on', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}']

Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats'
2 jobs: ['7628727', '7628580']
suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}']

Dead: None
1 jobs: ['7628793']

Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
3 jobs: ['7628834', '7628589', '7628712']
suites intersection: ['task/test_cephadm}']
suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']

Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh
1 jobs: ['7628587']
suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}']

Failure: valgrind error: Leak_StillReachable
operator new[](unsigned long)
UnknownInlinedFun
UnknownInlinedFun
3 jobs: ['7628743', '7628656', '7628753']
suites intersection: []
suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']

Failure: "2024-03-29T06:20:00.000190+0000 mon.a (mon.0) 1909 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log
1 jobs: ['7628646']
suites: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/reef', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/default', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}']

Dead: b'2024-03-29T18:59:25.254 DEBUG:teuthology.exit:Finished running handlers'
2 jobs: ['7628791', '7628640']
suites intersection: ['rados', 'rados/singleton-nomsgr/{all/admin_socket_output']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}']

Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh'
1 jobs: ['7628612']
suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}']

Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'
1 jobs: ['7628602']
suites: ['agent/off', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}']