Found 319 jobs Found 12 distinct failure reasons Failure: {'smithi045.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 11 jobs: ['6900917', '6901015', '6900970', '6901009', '6900851', '6900904', '6900926', '6900923', '6901069', '6901005', '6901040'] suites intersection: [] suites union: ['0-nvme-loop', '1-pg-log-overrides/short_pg_log', '1-start', '2-ops/rm-zap-add}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'agent/off', 'backoff/peering', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'fast/fast', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'rados/singleton-nomsgr/{all/ceph-post-file', 'rados/singleton/{all/divergent_priors2', 'rados/singleton/{all/ec-lost-unfound', 'rados/singleton/{all/test-crash', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'start}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'thrashers/default', 'thrashers/fastread', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/dedup-io-mixed}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aeb336ccdef47f3aab7c94484b6a2ca19fc47b14 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 3 jobs: ['6900899', '6900774', '6901011'] suites intersection: ['task/test_cephadm}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{agent/off', 'rados/cephadm/workunits/{agent/on', 'task/test_cephadm}'] Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS) 3 jobs: ['6901079', '6900963', '6900835'] suites intersection: ['task/test_nfs}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{agent/off', 'rados/cephadm/workunits/{agent/on', 'task/test_nfs}'] Failure: Command failed (workunit test cls/test_cls_rgw.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh' 2 jobs: ['6900872', '6901029'] suites intersection: ['0-start', '1-tasks', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['6900908'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] Failure: Test failure: test_perf_counters_list (tasks.mgr.dashboard.test_perf_counters.PerfCountersControllerTest) 1 jobs: ['6901026'] suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/dashboard}'] Crash: Command crashed: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir $TESTDIR/kvtest && cd $TESTDIR/kvtest && ceph_test_keyvaluedb'" ceph version 17.0.0-13236-gaeb336cc (aeb336ccdef47f3aab7c94484b6a2ca19fc47b14) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9708d18420] 2: gsignal() 3: abort() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x22729) [0x7f970793c729] 5: /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6) [0x7f970794dfd6] 6: ceph_test_keyvaluedb(+0x801135) [0x55e960b81135] 7: (KVTest_OpenClose_Test::TestBody()+0x23a) [0x55e960b81a0a] 8: (void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*)+0x51) [0x55e960bef941] 9: (testing::Test::Run()+0xd6) [0x55e960be2566] 10: (testing::TestInfo::Run()+0x135) [0x55e960be26c5] 11: (testing::TestSuite::Run()+0xc9) [0x55e960be2c59] 12: (testing::internal::UnitTestImpl::RunAllTests()+0x487) [0x55e960be3287] 13: (bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x51) [0x55e960befeb1] 14: (testing::UnitTest::Run()+0xa1) [0x55e960be2791] 15: main() 16: __libc_start_main() 17: _start() 1 jobs: ['6901044'] suites: ['rados/objectstore/{backends/keyvaluedb', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test cephtool/test.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aeb336ccdef47f3aab7c94484b6a2ca19fc47b14 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 4 jobs: ['6901016', '6900784', '6900939', '6900859'] suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}'] Failure: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds 4 jobs: ['6900788', '6900948', '6901027', '6900869'] suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04'] suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}'] Failure: Command failed (workunit test mon/test_mon_osdmap_prune.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aeb336ccdef47f3aab7c94484b6a2ca19fc47b14 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh' 1 jobs: ['6901064'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'thrashers/many', 'workloads/rados_mon_osdmap_prune}'] Failure: Command failed (workunit test osd/repro_long_log.sh) on smithi109 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aeb336ccdef47f3aab7c94484b6a2ca19fc47b14 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh' 1 jobs: ['6900800'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd}'] Crash: "2022-06-27T16:40:14.344915+0000 mon.b (mon.1) 89 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log ceph version 17.0.0-13236-gaeb336cc (aeb336ccdef47f3aab7c94484b6a2ca19fc47b14) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f0e79887ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f0e7be7c512] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2c16d5) [0x7f0e7be7c6d5] 6: (PaxosService::propose_pending()+0x50d) [0x55ea14fbb3ed] 7: ceph-mon(+0x4674f9) [0x55ea14fbb4f9] 8: (Context::complete(int)+0xd) [0x55ea14e7bced] 9: (CommonSafeTimer::timer_thread()+0x11a) [0x7f0e7bf90aba] 10: (CommonSafeTimerThread::entry()+0x11) [0x7f0e7bf920e1] 11: /lib64/libpthread.so.0(+0x81ca) [0x7f0e7987d1ca] 12: clone() 1 jobs: ['6900866'] suites: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zstd}', 'supported-random-distro$/{centos_8}', 'tasks/crash}']