Found 200 jobs Misunderstood line: Missing teuthology log /home/teuthworker/archive/akupczyk-2023-11-14_14:37:53-rados-aclamk-test-sb-to-b-vselector-distro-default-smithi/7457260/teuthology.log Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f401da54db0] , 2: (ceph::common::RefCountedObject::_get() const+0x26) [0x7f401ea55cd6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f401da54db0] , 2: (ceph::common::RefCountedObject::_get() const+0x26) [0x7f401ea55cd6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f401da54db0] , 2: (ceph::common::RefCountedObject::_get() const+0x26) [0x7f401ea55cd6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f401da54db0] , 2: (ceph::common::RefCountedObject::_get() const+0x26) [0x7f401ea55cd6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f401da54db0] , 2: (ceph::common::RefCountedObject::_get() const+0x26) [0x7f401ea55cd6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f401da54db0] , 2: (ceph::common::RefCountedObject::_get() const+0x26) [0x7f401ea55cd6] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Misunderstood line: Misunderstood line: Found 15 distinct failure reasons Dead: b'2023-11-15T02:49:33.579 DEBUG:teuthology.exit:Finished running handlers' 42 jobs suites intersection: ['rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/dedup-io-mixed}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/redirect_promote_tests}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] Valgrind: mon (Leak_StillReachable) 3 jobs: ['7457321', '7457306', '7457191'] suites intersection: [] suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] Crash: timed out waiting for admin_socket to appear after osd.1 restart ceph version 18.0.0-6888-g85686a58 (85686a58dd2879960d235ce0f01c2a7a8a70a45a) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f45f3c54db0] 2: (ceph::common::RefCountedObject::_get() const+0x24) [0x55fc76524bc4] 3: ceph-osd(+0x97a43c) [0x55fc763d743c] 4: (BlueStore::ExtentMap::ExtentDecoder::decode_extent(BlueStore::Extent*, unsigned char, ceph::buffer::v15_2_0::ptr::iterator_impl&, BlueStore::Collection*)+0x2ce) [0x55fc763ed73e] 5: (BlueStore::ExtentMap::ExtentDecoder::decode_some(ceph::buffer::v15_2_0::list const&, BlueStore::Collection*)+0x138) [0x55fc763ed928] 6: (BlueStore::Onode::decode_raw(BlueStore::Onode*, ceph::buffer::v15_2_0::list const&, BlueStore::ExtentMap::ExtentDecoder&)+0xc18) [0x55fc763fe108] 7: (BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0xdb3) [0x55fc76483113] 8: (BlueStore::reconstruct_allocations(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0x5a) [0x55fc7648403a] 9: (BlueStore::read_allocation_from_drive_on_startup()+0x10a) [0x55fc7648425a] 10: (BlueStore::_init_alloc(std::map, std::allocator > >*)+0x7c0) [0x55fc76408460] 11: (BlueStore::_open_db_and_around(bool, bool)+0x3e4) [0x55fc76410ae4] 12: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x18a) [0x55fc7642acba] 13: (BlueStore::_mount()+0x149) [0x55fc764169a9] 14: (OSD::init()+0x542) [0x55fc76017d92] 15: main() 16: /lib64/libc.so.6(+0x3feb0) [0x7f45f3c3feb0] 17: __libc_start_main() 18: _start() 19 jobs: ['7457219', '7457149', '7457225', '7457212', '7457196', '7457162', '7457325', '7457136', '7457332', '7457246', '7457171', '7457253', '7457188', '7457266', '7457168', '7457230', '7457288', '7457229', '7457183'] suites intersection: [] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/backfill-toofull', 'rados/singleton/{all/divergent_priors', 'rados/singleton/{all/divergent_priors2', 'rados/singleton/{all/dump-stuck', 'rados/singleton/{all/ec-inconsistent-hinfo', 'rados/singleton/{all/ec-lost-unfound', 'rados/singleton/{all/lost-unfound', 'rados/singleton/{all/lost-unfound-delete', 'rados/singleton/{all/osd-backfill', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/standalone/{supported-random-distro$/{centos_latest}', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'workloads/ec-small-objects-many-deletes}', 'workloads/ec-small-objects}', 'workloads/osd}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}'] Failure: ceph version 17.2.7-61.gb26a8038 was not installed, found 17.2.7-56.g0be64d43.el8. 1 jobs: ['7457206'] suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 1 jobs: ['7457240'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli}'] Dead: None 3 jobs: ['7457260', '7457154', '7457309'] Failure: 'pg2json' 1 jobs: ['7457202'] suites: ['rados/objectstore/{backends/ceph_objectstore_tool', 'supported-random-distro$/{centos_latest}}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['7457158', '7457284'] suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'supported-random-distro$/{centos_latest}', 'thrashosds-health', 'workloads/rados_api_tests}'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashers/mapgap', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=85686a58dd2879960d235ce0f01c2a7a8a70a45a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 1 jobs: ['7457290'] suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zstd}', 'tasks/e2e}'] Dead: b'OSError: Socket is closed' 1 jobs: ['7457331'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] Failure: reached maximum tries (801) after waiting for 4800 seconds 1 jobs: ['7457228'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=85686a58dd2879960d235ce0f01c2a7a8a70a45a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 1 jobs: ['7457330'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 1 jobs: ['7457297'] suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed on smithi192 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:85686a58dd2879960d235ce0f01c2a7a8a70a45a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 760e43e4-838f-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *'" 1 jobs: ['7457248'] suites: ['0-nvme-loop', 'agent/off', 'fixed-2', 'mon_election/connectivity', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'start}'] Dead: b'2023-11-15T20:46:36.161 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight' 1 jobs: ['7457334'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}']