Found 328 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-01_21:56:37-rados-wip-yuri5-testing-2024-01-31-1657-distro-default-smithi/7543071/teuthology.log Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-01_21:56:37-rados-wip-yuri5-testing-2024-01-31-1657-distro-default-smithi/7543071/teuthology.log Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-01_21:56:37-rados-wip-yuri5-testing-2024-01-31-1657-distro-default-smithi/7543071/teuthology.log Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-01_21:56:37-rados-wip-yuri5-testing-2024-01-31-1657-distro-default-smithi/7542965/teuthology.log Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-01_21:56:37-rados-wip-yuri5-testing-2024-01-31-1657-distro-default-smithi/7542965/teuthology.log Found 17 distinct failure reasons Dead: b'2024-02-02T16:39:07.538 DEBUG:teuthology.exit:Finished running handlers' 44 jobs suites intersection: ['rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-many-deletes}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/ec-snaps-few-objects-overwrites}'] Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e10f4b28ec4e5fd19161c663ab66bd3d25b667cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 1 jobs: ['7543037'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_timeout}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5571c0abbf85] ceph version 19.0.0-1243-ge10f4b28 (e10f4b28ec4e5fd19161c663ab66bd3d25b667cc) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f0798e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f0798ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5571c0abbf85] 6: ceph-osd(+0x3f30e6) [0x5571c0abc0e6] 7: ceph-osd(+0x39b274) [0x5571c0a64274] 8: (ECBackend::RecoveryBackend::continue_recovery_op(ECBackend::RecoveryBackend::RecoveryOp&, RecoveryMessages*)+0x129f) [0x5571c0fad9ff] 9: (ECBackend::RecoveryBackend::run_recovery_op(ECRecoveryHandle&, int)+0xf83) [0x5571c0fb0123] 10: (ECBackend::run_recovery_op(PGBackend::RecoveryHandle*, int)+0x23) [0x5571c0fb0533] 11: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x2845) [0x5571c0d70fe5] 12: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xf89) [0x5571c0d62de9] 13: (OSD::do_recovery(PG*, unsigned int, unsigned long, int, ThreadPool::TPHandle&)+0x245) [0x5571c0c46375] 14: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xca) [0x5571c0e91bba] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5571c0c626f7] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5571c1160a8a] 17: ceph-osd(+0xa98034) [0x5571c1161034] 18: /lib64/libc.so.6(+0x9f802) [0x7f0798e9f802] 19: /lib64/libc.so.6(+0x3f450) [0x7f0798e3f450] 1 jobs: ['7542849'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{centos_latest}}'] Failure: reached maximum tries (51) after waiting for 300 seconds 2 jobs: ['7543151', '7543032'] suites intersection: [] suites union: ['2-inject-leak/osd', 'centos_latest}', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'rados/valgrind-leaks/{1-start', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-radosbench}'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['7542860', '7543076'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] Timeout 120 running ceph --cluster ceph osd dump --format=json 2 jobs: ['7542843', '7543144'] suites intersection: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/repair_test}'] suites union: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/repair_test}'] Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 5 jobs: ['7542893', '7543041', '7543114', '7542970', '7543040'] suites intersection: [] suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] Timeout 3h running clone.client.0/qa/standalone/osd/divergent-priors.sh 1 jobs: ['7542865'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd}'] Failure: Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e10f4b28ec4e5fd19161c663ab66bd3d25b667cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh' 1 jobs: ['7543160'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd-backfill}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5592a1cee09c] ceph version 19.0.0-1243-ge10f4b28 (e10f4b28ec4e5fd19161c663ab66bd3d25b667cc) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd8e6238520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5592a1cee09c] 6: ceph-osd(+0x3f61fd) [0x5592a1cee1fd] 7: ceph-osd(+0x39f004) [0x5592a1c97004] 8: (ECBackend::RecoveryBackend::continue_recovery_op(ECBackend::RecoveryBackend::RecoveryOp&, RecoveryMessages*)+0x12cf) [0x5592a21cb71f] 9: (ECBackend::RecoveryBackend::run_recovery_op(ECRecoveryHandle&, int)+0xe4c) [0x5592a21cdbbc] 10: (ECBackend::run_recovery_op(PGBackend::RecoveryHandle*, int)+0x23) [0x5592a21cdfc3] 11: (PrimaryLogPG::recover_replicas(unsigned long, ThreadPool::TPHandle&, bool*)+0x12f2) [0x5592a1f88f52] 12: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x25b) [0x5592a1f9499b] 13: (OSD::do_recovery(PG*, unsigned int, unsigned long, int, ThreadPool::TPHandle&)+0x245) [0x5592a1e5fd85] 14: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xca) [0x5592a20bec3a] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5592a1e81543] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5592a237a6a3] 17: ceph-osd(+0xa82c04) [0x5592a237ac04] 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd8e628ab43] 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd8e631ca00] 1 jobs: ['7542901'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton/{all/ec-lost-unfound', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e10f4b28ec4e5fd19161c663ab66bd3d25b667cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 1 jobs: ['7542899'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['7542904', '7543082'] suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'supported-random-distro$/{centos_latest}', 'thrashosds-health', 'workloads/rados_api_tests}'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/read', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: Command failed on smithi106 with status 5: 'sudo systemctl stop ceph-088a7ef6-c187-11ee-95b6-87774f69a715@mon.b' 1 jobs: ['7542969'] suites: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/nautilus-v1only', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}'] Dead: None 2 jobs: ['7543071', '7542965'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2 jobs: ['7542862', '7543028'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: wait_for_clean: failed before timeout expired 1 jobs: ['7542891'] suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/ec-inconsistent-hinfo', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test erasure-code/test-erasure-code.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e10f4b28ec4e5fd19161c663ab66bd3d25b667cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh' 1 jobs: ['7542999'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/erasure-code}']