2024-01-20T07:23:34.213 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T07:23:34.213 INFO:teuthology.results:subset: '111/120000' 2024-01-20T07:23:34.213 INFO:teuthology.results:seed: '587' 2024-01-20T21:56:24.942 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T21:56:24.966 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-21T04:05:57.645 INFO:teuthology.results:Tests finished! gathering results... 2024-01-21T04:06:22.711 INFO:scrape:Found 317 jobs 2024-01-21T04:06:27.198 INFO:scrape:Found 9 distinct failure reasons 2024-01-21T04:06:27.198 INFO:scrape:Failure: No module named 'tasks' 2024-01-21T04:06:27.198 INFO:scrape:3 jobs: ['7522704', '7522688', '7522702'] 2024-01-21T04:06:27.198 INFO:scrape:suites intersection: ['msgr/async-v1only', 'rados'] 2024-01-21T04:06:27.199 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'd-thrash/default/{default', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/rados_cls_all', 'thrashers/morepggrow', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'workloads/small-objects-balanced}'] 2024-01-21T04:06:27.199 INFO:scrape: 2024-01-21T04:06:27.199 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-01-21T04:06:27.199 INFO:scrape:2 jobs: ['7523145', '7522748'] 2024-01-21T04:06:27.199 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-01-21T04:06:27.199 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-01-21T04:06:27.199 INFO:scrape: 2024-01-21T04:06:27.199 INFO:scrape:Failure: Command failed on smithi110 with status 1: 'sudo yum -y install cephfs-java' 2024-01-21T04:06:27.199 INFO:scrape:2 jobs: ['7522694', '7522690'] 2024-01-21T04:06:27.199 INFO:scrape:suites intersection: ['mon_election/classic'] 2024-01-21T04:06:27.200 INFO:scrape:suites union: ['agent/off', 'ceph', 'clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'task/test_ca_signed_key}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2024-01-21T04:06:27.200 INFO:scrape: 2024-01-21T04:06:27.200 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T04:06:27.200 INFO:scrape:5 jobs: ['7522773', '7523143', '7522770', '7522396', '7522987'] 2024-01-21T04:06:27.200 INFO:scrape:suites intersection: [] 2024-01-21T04:06:27.200 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-01-21T04:06:27.200 INFO:scrape: 2024-01-21T04:06:27.200 INFO:scrape:Crash: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T04:06:27.200 INFO:scrape:ceph version 19.0.0-763-ga00c9940 (a00c9940991c5e1e9ab595faa3f6e1281d8ae167) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb8b6cf] 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x72) [0xb8b7c2] 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x297) [0xb8f347] 6: ceph-osd(+0xa87904) [0xb8f904] 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-01-21T04:06:27.200 INFO:scrape:1 jobs: ['7522621'] 2024-01-21T04:06:27.201 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2024-01-21T04:06:27.201 INFO:scrape: 2024-01-21T04:06:27.201 INFO:scrape:Dead: b'2024-01-21T04:01:48.869 DEBUG:teuthology.exit:Finished running handlers' 2024-01-21T04:06:27.201 INFO:scrape:2 jobs: ['7522939', '7522510'] 2024-01-21T04:06:27.201 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/admin_socket_output'] 2024-01-21T04:06:27.201 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-01-21T04:06:27.201 INFO:scrape: 2024-01-21T04:06:27.201 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-01-21T04:06:27.201 INFO:scrape:1 jobs: ['7523101'] 2024-01-21T04:06:27.201 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-01-21T04:06:27.202 INFO:scrape: 2024-01-21T04:06:27.202 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2024-01-21T04:06:27.202 INFO:scrape:1 jobs: ['7522766'] 2024-01-21T04:06:27.202 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_timeout}'] 2024-01-21T04:06:27.202 INFO:scrape: 2024-01-21T04:06:27.202 INFO:scrape:Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2024-01-21T04:06:27.202 INFO:scrape:1 jobs: ['7522559'] 2024-01-21T04:06:27.202 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'task/test_orch_cli}'] 2024-01-21T04:06:27.202 INFO:scrape: 2024-01-21T04:06:27.213 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/ failed: 16 dead: 2 running: 0 waiting: 0 queued: 0 passed: 299 Fail ================================================================= [7522396] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:46:26 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522396/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522396/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7522559] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:22:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522559/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522559/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d69160b2ae494e0cb400107f3506ae5b Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7522621] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:30:01 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95d0a83841994618ab70022ecc08a6a1 valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7522688] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:07:51 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522688/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522688/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e349c4aa1834b8eaa330dec62b11926 No module named 'tasks.thrashosds' [7522690] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:10:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522690/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522690/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7cddcbe4a8b04b978e101358789b9d3f Command failed on smithi087 with status 1: 'sudo yum -y install cephfs- java' [7522694] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:05:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522694/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89a64bf23b59430fa73472d14d1b9e02 Command failed on smithi110 with status 1: 'sudo yum -y install cephfs- java' [7522702] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:08:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522702/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23d82c9731634121b149155ef19bda12 No module named 'tasks' [7522704] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:06:27 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522704/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=381d6b5fa891450ea5b2a1b0c5bc1383 No module named 'tasks' [7522748] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522748/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8976e2f7cf24ba0ae01ddac130d6e76 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7522766] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:15:22 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522766/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e264b4e783914341a1f8627610a5e0fb Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7522770] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:18:59 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522770/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522770/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7522773] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:30:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522773/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522773/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7522987] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:42:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522987/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522987/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7523101] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:14:20 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523101/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b25e4eefa6148a1980c2b3778f0a7a4 Command failed (workunit test rados/test.sh) on smithi055 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7523143] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:52:10 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523143/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523143/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7523145] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523145/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523145/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a3dfbdd191845079a09d574b4766884 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' Dead ================================================================= [7522510] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522510/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522510/ hit max job timeout [7522939] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522939/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522939/ hit max job timeout Pass ================================================================= [7522390] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:21:38 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522390/ [7522393] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:11:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522393/ [7522399] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:19:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522399/ [7522402] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522402/ [7522405] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:27 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522405/ [7522408] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522408/ [7522411] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 03:26:13 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522411/ [7522414] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:08:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522414/ [7522417] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:36:42 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522417/ [7522420] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/dedup-io-mixed} time: 00:13:20 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522420/ [7522423] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522423/ [7522426] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:22:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522426/ [7522429] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522429/ [7522432] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_host_drain} time: 00:22:23 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522432/ [7522435] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:30:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522435/ [7522438] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:09:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522438/ [7522441] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522441/ [7522444] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522444/ [7522447] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522447/ [7522450] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:23:55 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522450/ [7522453] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:14:23 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522453/ [7522456] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:33:48 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522456/ [7522459] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:14:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522459/ [7522462] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522462/ [7522465] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:31:26 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522465/ [7522468] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:27:38 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522468/ [7522471] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:23:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522471/ [7522474] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:24 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522474/ [7522477] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:13:14 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522477/ [7522480] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:33:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522480/ [7522483] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:16 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522483/ [7522486] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:30:20 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522486/ [7522489] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:29:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522489/ [7522492] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:20:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522492/ [7522495] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:27:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522495/ [7522498] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/dashboard} time: 01:22:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522498/ [7522501] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:23 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522501/ [7522504] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:39:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522504/ [7522507] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:09:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522507/ [7522513] rados/standalone/{supported-random-distro$/{centos_latest} workloads/c2c} time: 00:13:29 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522513/ [7522516] rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:45:59 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522516/ [7522519] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:16:08 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522519/ [7522522] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:11:13 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522522/ [7522525] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:41 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522525/ [7522528] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:09:45 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522528/ [7522531] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522531/ [7522535] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:46:18 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522535/ [7522541] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:45 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522541/ [7522544] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:17:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522544/ [7522547] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:41 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522547/ [7522550] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} time: 00:34:14 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522550/ [7522553] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:13:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522553/ [7522556] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:21:34 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522556/ [7522562] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:07:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522562/ [7522565] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:16 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522565/ [7522568] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522568/ [7522571] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/radosbench} time: 00:23:48 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522571/ [7522574] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:05 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522574/ [7522577] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:12:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522577/ [7522579] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:48 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522579/ [7522582] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522582/ [7522585] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:31:12 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522585/ [7522588] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522588/ [7522591] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:30:08 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522591/ [7522594] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:09:45 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522594/ [7522597] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:25 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522597/ [7522600] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:14:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522600/ [7522603] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522603/ [7522606] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:23:25 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522606/ [7522609] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522609/ [7522612] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:06:51 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522612/ [7522615] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:16:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522615/ [7522618] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:33:32 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522618/ [7522624] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:19:18 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522624/ [7522627] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:16:52 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522627/ [7522630] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:09:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522630/ [7522633] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522633/ [7522636] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:16:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522636/ [7522639] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:08 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522639/ [7522642] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:11:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522642/ [7522645] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:13:22 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522645/ [7522648] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_latest}} time: 00:06:52 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522648/ [7522651] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:18:32 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522651/ [7522654] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:16:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522654/ [7522657] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:58 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522657/ [7522660] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:22:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522660/ [7522663] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:23:11 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522663/ [7522666] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_adoption} time: 00:11:10 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522666/ [7522668] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:15:11 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522668/ [7522670] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:21:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522670/ [7522672] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522672/ [7522674] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522674/ [7522676] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:10 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522676/ [7522678] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:01 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522678/ [7522680] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:20:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522680/ [7522682] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:22:05 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522682/ [7522684] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:15:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522684/ [7522686] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:19:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522686/ [7522692] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:30:24 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522692/ [7522696] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:59 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522696/ [7522698] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:18 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522698/ [7522700] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:30:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522700/ [7522706] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:18:20 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522706/ [7522708] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:29 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522708/ [7522710] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm} time: 00:19:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522710/ [7522712] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:19:34 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522712/ [7522714] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522714/ [7522716] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:13:58 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522716/ [7522718] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522718/ [7522720] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:11 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522720/ [7522722] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/e2e} time: 00:23:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522722/ [7522724] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:18:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522724/ [7522726] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522726/ [7522728] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:14 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522728/ [7522730] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos} time: 00:07:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522730/ [7522732] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:07:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522732/ [7522734] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:25 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522734/ [7522736] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:09:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522736/ [7522738] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522738/ [7522740] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:22:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522740/ [7522742] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/rgw 3-final} time: 00:12:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522742/ [7522744] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:25:37 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522744/ [7522746] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} time: 00:11:38 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522746/ [7522750] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} time: 00:13:55 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522750/ [7522752] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:17:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522752/ [7522754] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:17:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522754/ [7522756] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:46:15 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522756/ [7522758] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:11:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522758/ [7522760] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:54 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522760/ [7522762] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:08 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522762/ [7522764] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522764/ [7522768] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:17:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522768/ [7522776] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522776/ [7522779] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:40:18 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522779/ [7522782] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522782/ [7522785] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:14:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522785/ [7522788] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:26:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522788/ [7522791] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:32:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522791/ [7522794] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_extra_daemon_features} time: 00:14:41 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522794/ [7522797] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:07:58 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522797/ [7522800] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:31:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522800/ [7522803] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:17:12 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522803/ [7522806] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_host_drain} time: 00:23:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522806/ [7522809] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:25:41 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522809/ [7522812] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:12:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522812/ [7522815] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:10:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522815/ [7522819] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:41 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522819/ [7522821] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:19:42 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522821/ [7522824] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522824/ [7522827] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522827/ [7522830] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:17:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522830/ [7522833] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:19:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522833/ [7522836] rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} time: 01:03:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522836/ [7522839] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:26 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522839/ [7522842] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522842/ [7522845] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:33:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522845/ [7522848] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:18:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522848/ [7522851] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522851/ [7522854] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:19:27 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522854/ [7522858] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522858/ [7522861] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:22:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522861/ [7522864] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:22:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522864/ [7522867] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:21:01 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522867/ [7522870] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:12:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522870/ [7522873] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:24:13 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522873/ [7522876] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:25:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522876/ [7522879] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:29 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522879/ [7522882] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:29 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522882/ [7522885] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:13:54 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522885/ [7522888] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:12:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522888/ [7522891] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:15:24 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522891/ [7522894] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:11:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522894/ [7522897] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:30:52 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522897/ [7522900] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:21:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522900/ [7522903] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522903/ [7522906] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:24 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522906/ [7522909] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522909/ [7522912] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:58:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522912/ [7522915] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:18:22 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522915/ [7522918] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:53 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522918/ [7522921] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:21:16 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522921/ [7522924] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli} time: 00:14:19 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522924/ [7522927] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:13:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522927/ [7522930] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:11 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522930/ [7522933] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:32:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522933/ [7522936] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:22:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522936/ [7522942] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:25:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522942/ [7522945] rados/upgrade/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:56:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522945/ [7522948] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:22:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522948/ [7522951] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:09:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522951/ [7522954] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:25:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522954/ [7522957] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522957/ [7522960] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/basic 3-final} time: 00:14:54 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522960/ [7522963] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:05 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522963/ [7522966] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:44 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522966/ [7522969] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:35 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522969/ [7522972] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:10:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522972/ [7522975] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522975/ [7522978] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:21:23 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522978/ [7522981] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522981/ [7522984] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522984/ [7522990] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:17:51 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522990/ [7522993] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:12:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522993/ [7522996] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 01:57:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522996/ [7522999] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:27:11 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522999/ [7523002] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_rgw_multisite} time: 00:22:16 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523002/ [7523005] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:18:22 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523005/ [7523008] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/readwrite} time: 00:10:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523008/ [7523011] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:11 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523011/ [7523014] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:10 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523014/ [7523017] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:47:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523017/ [7523019] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:21:42 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523019/ [7523021] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} time: 00:27:53 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523021/ [7523023] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:10:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523023/ [7523025] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:10:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523025/ [7523027] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:23 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523027/ [7523029] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:01 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523029/ [7523031] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:51 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523031/ [7523033] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:33:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523033/ [7523035] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_adoption} time: 00:10:25 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523035/ [7523038] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:19:14 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523038/ [7523040] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:19:19 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523040/ [7523042] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523042/ [7523044] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:53 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523044/ [7523046] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523046/ [7523048] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:03 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523048/ [7523050] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:22:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523050/ [7523051] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:18:29 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523051/ [7523053] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523053/ [7523056] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/repair_test} time: 00:14:33 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523056/ [7523057] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:13:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523057/ [7523059] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:08:29 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523059/ [7523061] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523061/ [7523063] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:12:42 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523063/ [7523066] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:16:37 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523066/ [7523068] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:54 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523068/ [7523070] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:18:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523070/ [7523072] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:11:34 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523072/ [7523074] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:07:42 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523074/ [7523075] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm} time: 00:15:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523075/ [7523077] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:21 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523077/ [7523079] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:26:55 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523079/ [7523081] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:23 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523081/ [7523083] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:09:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523083/ [7523085] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:49:15 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523085/ [7523087] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:33:43 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523087/ [7523089] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:14:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523089/ [7523091] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523091/ [7523093] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523093/ [7523095] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_latest}} time: 00:06:18 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523095/ [7523097] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:20 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523097/ [7523099] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523099/ [7523103] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:21:24 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523103/ [7523105] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:12:44 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523105/ [7523107] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:10:34 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523107/ [7523109] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:20 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523109/ [7523111] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:55:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523111/ [7523113] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523113/ [7523115] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm_timeout} time: 00:20:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523115/ [7523117] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:12:08 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523117/ [7523119] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:22:15 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523119/ [7523121] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523121/ [7523123] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:19:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523123/ [7523125] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/e2e} time: 00:26:09 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523125/ [7523127] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523127/ [7523129] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:22:40 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523129/ [7523131] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:13:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523131/ [7523133] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:31 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523133/ [7523135] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:11:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523135/ [7523137] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:44 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523137/ [7523139] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:11:13 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523139/ [7523141] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:16:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523141/ [7523147] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:05 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523147/ [7523149] rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523149/ [7523151] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:54 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523151/ [7523153] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:27:54 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523153/ [7523155] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:50 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523155/ [7523157] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523157/ [7523159] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:17:58 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523159/ [7523161] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523161/ [7523163] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:45 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523163/ [7523165] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:09:28 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523165/ [7523167] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_host_drain} time: 00:16:49 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523167/ [7523169] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:08 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523169/ [7523171] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523171/ [7523173] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:12:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523173/ [7523175] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:07 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523175/ [7523177] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/failover} time: 00:12:26 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523177/ [7523179] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:13:51 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523179/ [7523181] rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:20:12 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523181/ [7523183] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:19:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523183/ [7523185] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:12 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523185/ [7523187] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:09:24 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523187/ [7523189] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:24:39 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523189/ [7523191] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:15:37 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523191/ [7523193] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:12:56 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523193/ [7523195] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:16:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523195/ [7523197] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523197/ [7523199] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:40:06 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7523199/