2023-10-17T03:02:00.239 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T03:02:00.262 INFO:teuthology.results:subset: '111/120000' 2023-10-17T03:02:00.262 INFO:teuthology.results:seed: '9009' 2023-10-17T17:10:43.222 INFO:root:teuthology version: 0.0.1.dev199+gada4f44 2023-10-17T17:10:43.261 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-18T05:10:49.119 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-10-18T05:11:12.630 INFO:scrape:Found 309 jobs 2023-10-18T05:11:13.198 WARNING:scrape:Misunderstood line: 2023-10-18T05:11:36.419 WARNING:scrape:Misunderstood line: 2023-10-18T05:11:43.248 WARNING:scrape:Misunderstood line: 2023-10-18T05:11:43.432 WARNING:scrape:Misunderstood line: 2023-10-18T05:11:45.706 WARNING:scrape:Misunderstood line: 2023-10-18T05:11:45.707 INFO:scrape:Found 18 distinct failure reasons 2023-10-18T05:11:45.708 INFO:scrape:Failure: Found coredumps on ubuntu@smithi081.front.sepia.ceph.com 2023-10-18T05:11:45.708 INFO:scrape:1 jobs: ['7429863'] 2023-10-18T05:11:45.709 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/snaps-few-objects}'] 2023-10-18T05:11:45.709 INFO:scrape: 2023-10-18T05:11:45.710 INFO:scrape:Dead: b'2023-10-17T20:20:58.550 DEBUG:teuthology.exit:Finished running handlers' 2023-10-18T05:11:45.710 INFO:scrape:42 jobs 2023-10-18T05:11:45.711 INFO:scrape:suites intersection: ['rados'] 2023-10-18T05:11:45.711 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/monthrash/{ceph', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/small-objects-balanced}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}'] 2023-10-18T05:11:45.712 INFO:scrape: 2023-10-18T05:11:45.712 INFO:scrape:Valgrind: mon (Leak_StillReachable) 2023-10-18T05:11:45.713 INFO:scrape:5 jobs: ['7429792', '7429928', '7429857', '7429858', '7429719'] 2023-10-18T05:11:45.713 INFO:scrape:suites intersection: [] 2023-10-18T05:11:45.713 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2023-10-18T05:11:45.714 INFO:scrape: 2023-10-18T05:11:45.714 INFO:scrape:Failure: "2023-10-17T14:30:00.000150+0000 mon.a (mon.0) 187 : cluster [WRN] overall HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log 2023-10-18T05:11:45.715 INFO:scrape:1 jobs: ['7429934'] 2023-10-18T05:11:45.715 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/readwrite}'] 2023-10-18T05:11:45.716 INFO:scrape: 2023-10-18T05:11:45.716 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2023-10-18T05:11:45.716 INFO:scrape:3 jobs: ['7429668', '7429845', '7429846'] 2023-10-18T05:11:45.717 INFO:scrape:suites intersection: [] 2023-10-18T05:11:45.717 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/objectstore/{backends/keyvaluedb', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2023-10-18T05:11:45.718 INFO:scrape: 2023-10-18T05:11:45.718 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55998efdb4b6] 2023-10-18T05:11:45.718 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f29eb054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f29eb0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55998efdb4b6] 6: ceph-osd(+0x46e617) [0x55998efdb617] 7: ceph-osd(+0x42a15e) [0x55998ef9715e] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55998f290492] 9: ceph-osd(+0x56b441) [0x55998f0d8441] 10: ceph-osd(+0xea9e4b) [0x55998fa16e4b] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55998f222ec8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55998f35cbaf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xb97) [0x55998f16e307] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x262) [0x55998f6380b2] 15: ceph-osd(+0xacb624) [0x55998f638624] 16: /lib64/libc.so.6(+0x9f802) [0x7f29eb09f802] 17: /lib64/libc.so.6(+0x3f450) [0x7f29eb03f450] 2023-10-18T05:11:45.719 INFO:scrape:1 jobs: ['7429905'] 2023-10-18T05:11:45.719 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}'] 2023-10-18T05:11:45.720 INFO:scrape: 2023-10-18T05:11:45.720 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55934ad684f7] 2023-10-18T05:11:45.720 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5400de1520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55934ad684f7] 6: ceph-osd(+0xcd4658) [0x55934ad68658] 7: ceph-osd(+0xc8fd62) [0x55934ad23d62] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55934afb74a2] 9: ceph-osd(+0xd7e9c1) [0x55934ae129c1] 10: ceph-osd(+0x1695efb) [0x55934b729efb] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55934af543d8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55934b089fbf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55934aea8fa3] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55934b359923] 15: ceph-osd(+0x12c5e54) [0x55934b359e54] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5400e33b43] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5400ec5a00] 2023-10-18T05:11:45.721 INFO:scrape:1 jobs: ['7429892'] 2023-10-18T05:11:45.721 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-10-18T05:11:45.722 INFO:scrape: 2023-10-18T05:11:45.722 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x561fa61b44b6] 2023-10-18T05:11:45.722 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fa32d054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fa32d0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x561fa61b44b6] 6: ceph-osd(+0x46e617) [0x561fa61b4617] 7: ceph-osd(+0x42a15e) [0x561fa617015e] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x561fa6469492] 9: ceph-osd(+0x56b441) [0x561fa62b1441] 10: ceph-osd(+0xea9e4b) [0x561fa6befe4b] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x561fa63fbec8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x561fa6535baf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xb97) [0x561fa6347307] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x262) [0x561fa68110b2] 15: ceph-osd(+0xacb624) [0x561fa6811624] 16: /lib64/libc.so.6(+0x9f802) [0x7fa32d09f802] 17: /lib64/libc.so.6(+0x3f450) [0x7fa32d03f450] 2023-10-18T05:11:45.723 INFO:scrape:1 jobs: ['7429827'] 2023-10-18T05:11:45.723 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-10-18T05:11:45.724 INFO:scrape: 2023-10-18T05:11:45.724 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c4ffa1e4f7] 2023-10-18T05:11:45.725 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7659f0f520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c4ffa1e4f7] 6: ceph-osd(+0xcd4658) [0x55c4ffa1e658] 7: ceph-osd(+0xc8fd62) [0x55c4ff9d9d62] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55c4ffc6d4a2] 9: ceph-osd(+0xd7e9c1) [0x55c4ffac89c1] 10: ceph-osd(+0x1695efb) [0x55c5003dfefb] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55c4ffc0a3d8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55c4ffd3ffbf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55c4ffb5efa3] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55c50000f923] 15: ceph-osd(+0x12c5e54) [0x55c50000fe54] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7659f61b43] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7659ff3a00] 2023-10-18T05:11:45.725 INFO:scrape:1 jobs: ['7429927'] 2023-10-18T05:11:45.725 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-10-18T05:11:45.726 INFO:scrape: 2023-10-18T05:11:45.726 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-10-18T05:11:45.727 INFO:scrape:2 jobs: ['7429758', '7429913'] 2023-10-18T05:11:45.727 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-10-18T05:11:45.727 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-10-18T05:11:45.728 INFO:scrape: 2023-10-18T05:11:45.728 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55bad26434f7] 2023-10-18T05:11:45.729 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb65a1f8520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55bad26434f7] 6: ceph-osd(+0xcd4658) [0x55bad2643658] 7: ceph-osd(+0xc8fd62) [0x55bad25fed62] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55bad28924a2] 9: ceph-osd(+0xd7e9c1) [0x55bad26ed9c1] 10: ceph-osd(+0x1695efb) [0x55bad3004efb] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55bad282f3d8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55bad2964fbf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55bad2783fa3] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55bad2c34923] 15: ceph-osd(+0x12c5e54) [0x55bad2c34e54] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb65a24ab43] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb65a2dca00] 2023-10-18T05:11:45.729 INFO:scrape:1 jobs: ['7429670'] 2023-10-18T05:11:45.729 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-10-18T05:11:45.730 INFO:scrape: 2023-10-18T05:11:45.730 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2023-10-18T05:11:45.731 INFO:scrape:1 jobs: ['7429853'] 2023-10-18T05:11:45.731 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] 2023-10-18T05:11:45.731 INFO:scrape: 2023-10-18T05:11:45.732 INFO:scrape:Failure: smithi186.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - ceph - ceph-base - ceph-selinux - ceph-common - ceph-debuginfo - ceph-release - libcephfs1 - ceph-radosgw - python-ceph - python-rados - python-rbd - python-cephfs - librbd1 - librados2 - mod_fastcgi releasever: null security: false skip_broken: false state: absent update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''powertools'': Yum repo downloading error: Downloading error(s): repodata/b3790f172deea54d71e93ab1e72eda8b8e31356b51edd645a960a887a599a6da-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce2b76c73062e056a4e8e738d20101a4b72c9a9692c612359b2c58e73d04c767-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: []smithi026.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - ceph - ceph-base - ceph-selinux - ceph-common - ceph-debuginfo - ceph-release - libcephfs1 - ceph-radosgw - python-ceph - python-rados - python-rbd - python-cephfs - librbd1 - librados2 - mod_fastcgi releasever: null security: false skip_broken: false state: absent update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''powertools'': Yum repo downloading error: Downloading error(s): repodata/b3790f172deea54d71e93ab1e72eda8b8e31356b51edd645a960a887a599a6da-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce2b76c73062e056a4e8e738d20101a4b72c9a9692c612359b2c58e73d04c767-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: [] 2023-10-18T05:11:45.732 INFO:scrape:2 jobs: ['7429895', '7429901'] 2023-10-18T05:11:45.733 INFO:scrape:suites intersection: ['0-nvme-loop'] 2023-10-18T05:11:45.733 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'agent/off', 'fixed-2', 'mon_election/classic', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'start}'] 2023-10-18T05:11:45.733 INFO:scrape: 2023-10-18T05:11:45.734 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5626ed5d14b6] 2023-10-18T05:11:45.734 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f92ed054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f92ed0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5626ed5d14b6] 6: ceph-osd(+0x46e617) [0x5626ed5d1617] 7: ceph-osd(+0x42a15e) [0x5626ed58d15e] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5626ed886492] 9: ceph-osd(+0x56b441) [0x5626ed6ce441] 10: ceph-osd(+0xea9e4b) [0x5626ee00ce4b] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5626ed818ec8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x5626ed952baf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xb97) [0x5626ed764307] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x262) [0x5626edc2e0b2] 15: ceph-osd(+0xacb624) [0x5626edc2e624] 16: /lib64/libc.so.6(+0x9f802) [0x7f92ed09f802] 17: /lib64/libc.so.6(+0x3f450) [0x7f92ed03f450] 2023-10-18T05:11:45.735 INFO:scrape:1 jobs: ['7429960'] 2023-10-18T05:11:45.735 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-10-18T05:11:45.735 INFO:scrape: 2023-10-18T05:11:45.736 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a5df55d22d3638066184c241442ff00fdc0ebdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-10-18T05:11:45.736 INFO:scrape:2 jobs: ['7429832', '7429989'] 2023-10-18T05:11:45.737 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] 2023-10-18T05:11:45.737 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zstd}', 'random-objectstore$/{bluestore-stupid}', 'tasks/e2e}'] 2023-10-18T05:11:45.737 INFO:scrape: 2023-10-18T05:11:45.738 INFO:scrape:Dead: b'2023-10-18T05:11:28.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops' 2023-10-18T05:11:45.738 INFO:scrape:1 jobs: ['7429991'] 2023-10-18T05:11:45.739 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-snaps-few-objects-overwrites}'] 2023-10-18T05:11:45.739 INFO:scrape: 2023-10-18T05:11:45.739 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x559c00bd64f7] 2023-10-18T05:11:45.740 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f940a9b6520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x559c00bd64f7] 6: ceph-osd(+0xcd4658) [0x559c00bd6658] 7: ceph-osd(+0xc8fd62) [0x559c00b91d62] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x559c00e254a2] 9: ceph-osd(+0xd7e9c1) [0x559c00c809c1] 10: ceph-osd(+0x1695efb) [0x559c01597efb] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x559c00dc23d8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x559c00ef7fbf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x559c00d16fa3] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x559c011c7923] 15: ceph-osd(+0x12c5e54) [0x559c011c7e54] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f940aa08b43] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f940aa9aa00] 2023-10-18T05:11:45.740 INFO:scrape:1 jobs: ['7429902'] 2023-10-18T05:11:45.741 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/fastclose', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-small-objects-fast-read-overwrites}'] 2023-10-18T05:11:45.741 INFO:scrape: 2023-10-18T05:11:45.741 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5626af1dd4f7] 2023-10-18T05:11:45.742 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6d3f66e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5626af1dd4f7] 6: ceph-osd(+0xcd4658) [0x5626af1dd658] 7: ceph-osd(+0xc8fd62) [0x5626af198d62] 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5626af42c4a2] 9: ceph-osd(+0xd7e9c1) [0x5626af2879c1] 10: ceph-osd(+0x1695efb) [0x5626afb9eefb] 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5626af3c93d8] 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x5626af4fefbf] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5626af31dfa3] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x5626af7ce923] 15: ceph-osd(+0x12c5e54) [0x5626af7cee54] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f6d3f6c0b43] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f6d3f752a00] 2023-10-18T05:11:45.742 INFO:scrape:1 jobs: ['7429706'] 2023-10-18T05:11:45.743 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-snaps}'] 2023-10-18T05:11:45.743 INFO:scrape: 2023-10-18T05:11:45.755 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/ failed: 23 dead: 42 running: 3 waiting: 0 queued: 0 passed: 241 Fail ================================================================= [7429668] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429668/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429668/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c036f73690c4cda8ca51f12e7098dcf Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7429670] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:21:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429670/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429670/ Command failed on smithi046 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429706] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:29:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429706/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429706/ Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429719] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:44:26 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429719/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429719/ saw valgrind issues [7429758] rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:11:37 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429758/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2810cd98f5749b7acd9ab9fffaf7b8f Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi045 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7429792] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:45:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429792/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429792/ saw valgrind issues [7429827] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:19:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429827/ Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429832] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/e2e} ----------------------------------------------------------------- time: 00:30:43 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429832/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6851b1932ea84a92ba7fe687bd7aa291 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a5df55d22d3638066184c241442ff00fdc0ebdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7429845] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429845/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429845/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2717120856154a289d9c1b2bcb2499bf Command failed on smithi033 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir $TESTDIR/kvtest && cd $TESTDIR/kvtest && ceph_test_keyvaluedb'" [7429846] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:09:24 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429846/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a46c0b245d104d14814232fcec7c6b3a Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7429853] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:36:06 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429853/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429853/ Command failed on smithi123 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7429857] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:22:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429857/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429857/ saw valgrind issues [7429858] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:32:54 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429858/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429858/ saw valgrind issues [7429863] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:25:36 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429863/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429863/ Found coredumps on ubuntu@smithi081.front.sepia.ceph.com [7429892] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429892/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429892/ Found coredumps on ubuntu@smithi123.front.sepia.ceph.com [7429902] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:27:46 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429902/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429902/ Command failed on smithi094 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429905] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:25:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429905/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429905/ Command failed on smithi089 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429913] rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:13:36 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429913/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc5772db2b994d09bc3a624d012d0e60 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi139 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7429927] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:21:57 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429927/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429927/ Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429928] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:51:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429928/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429928/ saw valgrind issues [7429934] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/readwrite} ----------------------------------------------------------------- time: 00:11:54 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429934/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429934/ "2023-10-17T14:30:00.000150+0000 mon.a (mon.0) 187 : cluster [WRN] overall HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [7429960] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:19:33 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429960/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429960/ Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7429989] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/e2e} ----------------------------------------------------------------- time: 00:30:46 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429989/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d819acaecbf401d833357d46d051cf5 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a5df55d22d3638066184c241442ff00fdc0ebdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' Dead ================================================================= [7429661] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429661/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429661/ hit max job timeout [7429672] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429672/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429672/ hit max job timeout [7429690] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429690/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429690/ hit max job timeout [7429700] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429700/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429700/ hit max job timeout [7429712] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429712/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429712/ hit max job timeout [7429714] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429714/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429714/ hit max job timeout [7429728] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429728/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429728/ hit max job timeout [7429734] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429734/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429734/ hit max job timeout [7429736] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429736/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429736/ hit max job timeout [7429742] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429742/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429742/ hit max job timeout [7429744] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429744/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429744/ hit max job timeout [7429748] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429748/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429748/ hit max job timeout [7429756] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429756/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429756/ hit max job timeout [7429768] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429768/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429768/ hit max job timeout [7429776] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429776/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429776/ hit max job timeout [7429781] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429781/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429781/ hit max job timeout [7429784] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429784/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429784/ hit max job timeout [7429805] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429805/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429805/ hit max job timeout [7429814] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429814/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429814/ hit max job timeout [7429816] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429816/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429816/ hit max job timeout [7429817] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429817/ hit max job timeout [7429819] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429819/ hit max job timeout [7429834] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429834/ hit max job timeout [7429841] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429841/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429841/ hit max job timeout [7429844] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429844/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429844/ hit max job timeout [7429852] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429852/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429852/ hit max job timeout [7429859] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429859/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429859/ hit max job timeout [7429860] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429860/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429860/ hit max job timeout [7429875] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429875/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429875/ hit max job timeout [7429884] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429884/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429884/ hit max job timeout [7429895] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:11:46 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429895/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=311cf191a6164ab9906af0ff1539d371 smithi186.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - ceph - ceph-base - ceph-selinux - ceph-common - ceph-debuginfo - ceph-release - libcephfs1 - ceph-radosgw - python-ceph - python-rados - python-rbd - python-cephfs - librbd1 - librados2 - mod_fastcgi releasever: null security: false skip_broken: false state: absent update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''powertools'': Yum repo downloading error: Downloading error(s): repodata/b3790f172deea54d71e93ab1e72eda8b8e31356b51edd645a960a887a599a6da- primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce2b76c73062e056a4e8e738d20101a4b72c9a9692c612359b2c58e73 d04c767-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: []smithi026.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - ceph - ceph-base - ceph- selinux - ceph-common - ceph-debuginfo - ceph-release - libcephfs1 - ceph-radosgw - python-ceph - python-rados - python-rbd - python-cephfs - librbd1 - librados2 - mod_fastcgi releasever: null security: false skip_broken: false state: absent update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''powertools'': Yum repo downloading error: Downloading error(s): repodata/b3790f172deea54d71e93ab1e72eda8b8e31356b51edd645a960a887a599a6da- primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce2b76c73062e056a4e8e738d20101a4b72c9a9692c612359b2c58e73 d04c767-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: [] [7429901] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:49 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2995f397aaf746b08570dc140ff23d52 smithi081.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - ceph - ceph-base - ceph-selinux - ceph-common - ceph-debuginfo - ceph-release - libcephfs1 - ceph-radosgw - python-ceph - python-rados - python-rbd - python-cephfs - librbd1 - librados2 - mod_fastcgi releasever: null security: false skip_broken: false state: absent update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''powertools'': Yum repo downloading error: Downloading error(s): repodata/b3790f172deea54d71e93ab1e72eda8b8e31356b51edd645a960a887a599a6da- primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce2b76c73062e056a4e8e738d20101a4b72c9a9692c612359b2c58e73 d04c767-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: []smithi184.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - ceph - ceph-base - ceph- selinux - ceph-common - ceph-debuginfo - ceph-release - libcephfs1 - ceph-radosgw - python-ceph - python-rados - python-rbd - python-cephfs - librbd1 - librados2 - mod_fastcgi releasever: null security: false skip_broken: false state: absent update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''powertools'': Yum repo downloading error: Downloading error(s): repodata/b3790f172deea54d71e93ab1e72eda8b8e31356b51edd645a960a887a599a6da- primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce2b76c73062e056a4e8e738d20101a4b72c9a9692c612359b2c58e73 d04c767-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: [] [7429908] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429908/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429908/ hit max job timeout [7429911] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429911/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429911/ hit max job timeout [7429916] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429916/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429916/ hit max job timeout [7429919] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429919/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429919/ hit max job timeout [7429920] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429920/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429920/ hit max job timeout [7429933] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429933/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429933/ hit max job timeout [7429944] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429944/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429944/ hit max job timeout [7429951] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429951/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429951/ hit max job timeout [7429959] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429959/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429959/ hit max job timeout [7429968] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429968/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429968/ hit max job timeout Running ================================================================= [7429987] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429987/ [7429988] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429988/ [7429991] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429991/ Pass ================================================================= [7429659] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_ca_signed_key} time: 00:15:59 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429659/ [7429663] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:12:52 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429663/ [7429664] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/lockdep} time: 00:25:45 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429664/ [7429666] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/rgw 3-final} time: 00:13:35 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429666/ [7429674] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:33:32 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429674/ [7429676] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:09:58 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429676/ [7429678] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:19:28 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429678/ [7429680] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/scrub_test} time: 00:11:32 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429680/ [7429682] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:14:49 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429682/ [7429684] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:46:09 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429684/ [7429686] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:14:47 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429686/ [7429688] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_cephadm} time: 00:28:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429688/ [7429692] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:22 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429692/ [7429694] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:45:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429694/ [7429696] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:13:07 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429696/ [7429698] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429698/ [7429702] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:44 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429702/ [7429704] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:49 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429704/ [7429708] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:12:25 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429708/ [7429710] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:10:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429710/ [7429711] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/classic task/test_cephadm_repos} time: 00:11:35 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429711/ [7429713] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:21:23 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429713/ [7429715] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:17:59 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429715/ [7429716] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache} time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429716/ [7429717] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:15:46 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429717/ [7429718] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:13:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429718/ [7429720] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:14:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429720/ [7429721] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:19:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429721/ [7429722] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_extra_daemon_features} time: 00:19:56 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429722/ [7429723] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:09 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429723/ [7429724] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} time: 02:42:37 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429724/ [7429725] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:10:14 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429725/ [7429726] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:38:12 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429726/ [7429727] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/dedup-io-mixed} time: 00:14:13 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429727/ [7429729] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:04 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429729/ [7429730] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429730/ [7429731] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:30:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429731/ [7429732] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:42 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429732/ [7429733] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:25:11 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429733/ [7429735] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:48 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429735/ [7429737] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:26:44 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429737/ [7429738] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:23:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429738/ [7429739] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:32:04 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429739/ [7429740] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429740/ [7429741] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:27:03 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429741/ [7429743] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:21:29 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429743/ [7429745] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:14:04 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429745/ [7429746] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:24 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429746/ [7429747] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:34:40 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429747/ [7429749] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:30:28 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429749/ [7429750] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/connectivity task/test_orch_cli} time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429750/ [7429751] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:26:58 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429751/ [7429752] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} time: 01:37:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429752/ [7429753] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:14:03 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429753/ [7429754] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:41:38 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429754/ [7429755] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:12:53 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429755/ [7429757] rados/standalone/{supported-random-distro$/{centos_latest} workloads/c2c} time: 00:15:33 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429757/ [7429759] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:22 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429759/ [7429760] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:12:56 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429760/ [7429761] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429761/ [7429762] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:08:52 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429762/ [7429763] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429763/ [7429764] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:40:09 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429764/ [7429765] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:05 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429765/ [7429766] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429766/ [7429767] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final} time: 00:15:54 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429767/ [7429769] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:37 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429769/ [7429770] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} time: 00:16:26 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429770/ [7429771] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:38 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429771/ [7429772] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:42 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429772/ [7429773] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/classic task/test_orch_cli_mon} time: 00:25:54 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429773/ [7429774] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:11:13 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429774/ [7429775] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/radosbench} time: 00:25:03 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429775/ [7429777] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429777/ [7429778] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429778/ [7429779] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:31 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429779/ [7429780] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:12:45 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429780/ [7429782] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429782/ [7429783] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_rgw_multisite} time: 00:20:13 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429783/ [7429785] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/failover} time: 00:15:44 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429785/ [7429786] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:20:32 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429786/ [7429787] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:54 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429787/ [7429788] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429788/ [7429789] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:21:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429789/ [7429790] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:12:33 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429790/ [7429791] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:40:45 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429791/ [7429793] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:17:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429793/ [7429794] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:11:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429794/ [7429795] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:57 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429795/ [7429796] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_set_mon_crush_locations} time: 00:22:28 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429796/ [7429797] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:58 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429797/ [7429798] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429798/ [7429799] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429799/ [7429800] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:10:27 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429800/ [7429801] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429801/ [7429802] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:19:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429802/ [7429803] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:02 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429803/ [7429804] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:24:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429804/ [7429806] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_adoption} time: 00:15:41 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429806/ [7429807] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:12:06 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429807/ [7429808] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:19:08 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429808/ [7429809] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:37 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429809/ [7429810] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:49 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429810/ [7429811] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:52 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429811/ [7429812] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:16:20 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429812/ [7429813] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429813/ [7429815] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:15:23 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429815/ [7429818] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} time: 01:23:36 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429818/ [7429820] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_ca_signed_key} time: 00:16:08 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429820/ [7429821] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:06 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429821/ [7429822] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:31:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429822/ [7429823] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:23:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429823/ [7429824] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:18:08 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429824/ [7429825] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429825/ [7429826] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:19:44 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429826/ [7429828] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:34 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429828/ [7429829] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:15:06 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429829/ [7429830] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:18:28 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429830/ [7429831] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:55 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429831/ [7429833] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:47:31 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429833/ [7429835] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} time: 00:11:09 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429835/ [7429836] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:47 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429836/ [7429837] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/connectivity task/test_cephadm} time: 00:17:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429837/ [7429838] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:09:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429838/ [7429839] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:48 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429839/ [7429840] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:09:09 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429840/ [7429842] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:31:20 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429842/ [7429843] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:17:33 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429843/ [7429847] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:18:07 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429847/ [7429848] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:16:52 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429848/ [7429849] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/classic task/test_cephadm_repos} time: 00:09:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429849/ [7429850] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:40:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429850/ [7429851] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429851/ [7429854] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:12:25 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429854/ [7429855] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:16:24 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429855/ [7429856] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:16:25 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429856/ [7429861] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:23:47 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429861/ [7429862] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:15:31 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429862/ [7429864] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:34 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429864/ [7429865] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429865/ [7429866] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_extra_daemon_features} time: 00:20:51 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429866/ [7429867] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:19:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429867/ [7429868] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:30:00 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429868/ [7429869] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:33 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429869/ [7429870] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats} time: 00:09:38 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429870/ [7429871] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:23:44 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429871/ [7429872] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429872/ [7429873] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:18:47 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429873/ [7429874] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:36 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429874/ [7429876] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:49 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429876/ [7429877] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:22:23 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429877/ [7429878] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:16:32 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429878/ [7429879] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:12:55 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429879/ [7429880] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:14:12 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429880/ [7429881] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:32:34 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429881/ [7429882] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:22:27 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429882/ [7429883] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429883/ [7429885] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:33 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429885/ [7429886] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:23:27 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429886/ [7429887] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:58 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429887/ [7429888] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:19:20 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429888/ [7429889] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:13:12 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429889/ [7429890] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_orch_cli} time: 00:19:07 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429890/ [7429891] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:26:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429891/ [7429893] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:27:25 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429893/ [7429894] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:46 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429894/ [7429896] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:15:24 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429896/ [7429897] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:12:27 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429897/ [7429898] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429898/ [7429899] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/progress} time: 00:22:05 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429899/ [7429900] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:12:17 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429900/ [7429903] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:54:07 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429903/ [7429904] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:57 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429904/ [7429906] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_orch_cli_mon} time: 00:40:12 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429906/ [7429907] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:15:04 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429907/ [7429909] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:34:12 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429909/ [7429910] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} time: 01:44:23 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429910/ [7429912] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:27:48 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429912/ [7429914] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:23 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429914/ [7429915] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/basic 3-final} time: 00:15:02 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429915/ [7429917] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:59 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429917/ [7429918] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429918/ [7429921] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_rgw_multisite} time: 00:16:29 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429921/ [7429922] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:09:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429922/ [7429923] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429923/ [7429924] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:20:19 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429924/ [7429925] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:15:01 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429925/ [7429926] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:14:32 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429926/ [7429929] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429929/ [7429930] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 02:15:07 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429930/ [7429931] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:19:38 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429931/ [7429932] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} time: 02:22:29 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429932/ [7429935] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429935/ [7429936] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:15:03 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429936/ [7429937] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:53:56 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429937/ [7429938] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/classic task/test_set_mon_crush_locations} time: 00:23:11 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429938/ [7429939] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} time: 00:29:31 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429939/ [7429940] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:09:10 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429940/ [7429941] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:14:46 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429941/ [7429942] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:30 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429942/ [7429943] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:47 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429943/ [7429945] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_adoption} time: 00:15:57 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429945/ [7429946] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:23:05 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429946/ [7429947] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:18 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429947/ [7429948] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429948/ [7429949] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:13 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429949/ [7429950] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:13:03 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429950/ [7429952] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:25:13 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429952/ [7429953] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:20:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429953/ [7429954] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:20 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429954/ [7429955] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:18:58 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429955/ [7429956] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:48 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429956/ [7429957] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:10:16 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429957/ [7429958] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429958/ [7429961] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:23:21 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429961/ [7429962] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:12:56 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429962/ [7429963] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_ca_signed_key} time: 00:18:20 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429963/ [7429964] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429964/ [7429965] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:20:24 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429965/ [7429966] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:53 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429966/ [7429967] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:45 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429967/ [7429969] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:34 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429969/ [7429970] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} time: 02:34:08 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429970/ [7429971] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:35:01 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429971/ [7429972] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:19:20 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429972/ [7429973] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:02 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429973/ [7429974] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:37 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429974/ [7429975] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429975/ [7429976] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:23 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429976/ [7429977] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:15:08 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429977/ [7429978] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:27:22 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429978/ [7429979] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:22:44 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429979/ [7429980] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_cephadm} time: 00:18:50 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429980/ [7429981] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:39 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429981/ [7429982] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:59 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429982/ [7429983] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:27 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429983/ [7429984] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/classic task/test_cephadm_repos} time: 00:09:18 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429984/ [7429985] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:11:08 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429985/ [7429986] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:23:45 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429986/ [7429990] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/crash} time: 00:12:30 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429990/ [7429992] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:16:18 info: http://pulpito.ceph.com/yuriw-2023-10-16_14:44:27-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7429992/