2024-02-09T21:28:26.440 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-09T21:28:26.441 INFO:teuthology.results:subset: '111/120000' 2024-02-09T21:28:26.441 INFO:teuthology.results:seed: '2011' 2024-02-10T03:19:13.399 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-10T03:19:13.416 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-10T13:07:09.739 INFO:teuthology.results:Tests finished! gathering results... 2024-02-10T13:07:55.877 INFO:scrape:Found 325 jobs 2024-02-10T13:07:57.587 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554586/teuthology.log 2024-02-10T13:08:09.984 INFO:scrape:Found 14 distinct failure reasons 2024-02-10T13:08:09.985 INFO:scrape:Failure: "2024-02-10T02:57:29.780920+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-02-10T13:08:09.985 INFO:scrape:34 jobs 2024-02-10T13:08:10.015 INFO:scrape:suites intersection: [] 2024-02-10T13:08:10.015 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/pacific', '1-install/quincy', '1-install/reef', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'agent/off', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'start}', 'task/test_host_drain}', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2024-02-10T13:08:10.015 INFO:scrape: 2024-02-10T13:08:10.015 INFO:scrape:Failure: Command failed on smithi038 with status 1: 'sudo yum install -y kernel' 2024-02-10T13:08:10.015 INFO:scrape:1 jobs: ['7554418'] 2024-02-10T13:08:10.015 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_ca_signed_key}'] 2024-02-10T13:08:10.016 INFO:scrape: 2024-02-10T13:08:10.016 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-10T13:08:10.016 INFO:scrape:4 jobs: ['7554642', '7554643', '7554898', '7554751'] 2024-02-10T13:08:10.016 INFO:scrape:suites intersection: [] 2024-02-10T13:08:10.016 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-02-10T13:08:10.016 INFO:scrape: 2024-02-10T13:08:10.016 INFO:scrape:Dead: None 2024-02-10T13:08:10.016 INFO:scrape:8 jobs: ['7554586', '7554343', '7554410', '7554519', '7554655', '7554841', '7554876', '7554548'] 2024-02-10T13:08:10.016 INFO:scrape: 2024-02-10T13:08:10.017 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2024-02-10T13:08:10.017 INFO:scrape:1 jobs: ['7554745'] 2024-02-10T13:08:10.017 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_cephadm_timeout}'] 2024-02-10T13:08:10.017 INFO:scrape: 2024-02-10T13:08:10.017 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-02-10T13:08:10.017 INFO:scrape:2 jobs: ['7554629', '7554904'] 2024-02-10T13:08:10.017 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-02-10T13:08:10.017 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-02-10T13:08:10.017 INFO:scrape: 2024-02-10T13:08:10.017 INFO:scrape:Dead: b'2024-02-10T13:01:46.256 DEBUG:teuthology.exit:Finished running handlers' 2024-02-10T13:08:10.018 INFO:scrape:2 jobs: ['7554717', '7554444'] 2024-02-10T13:08:10.018 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}'] 2024-02-10T13:08:10.018 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}'] 2024-02-10T13:08:10.018 INFO:scrape: 2024-02-10T13:08:10.018 INFO:scrape:Failure: Failed to reconnect to smithi038 2024-02-10T13:08:10.018 INFO:scrape:1 jobs: ['7554426'] 2024-02-10T13:08:10.018 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect}'] 2024-02-10T13:08:10.018 INFO:scrape: 2024-02-10T13:08:10.018 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-02-10T13:08:10.018 INFO:scrape:1 jobs: ['7554388'] 2024-02-10T13:08:10.019 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-02-10T13:08:10.019 INFO:scrape: 2024-02-10T13:08:10.019 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-02-10T13:08:10.019 INFO:scrape:1 jobs: ['7554487'] 2024-02-10T13:08:10.019 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-02-10T13:08:10.019 INFO:scrape: 2024-02-10T13:08:10.019 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-02-10T13:08:10.019 INFO:scrape:1 jobs: ['7554369'] 2024-02-10T13:08:10.019 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-02-10T13:08:10.019 INFO:scrape: 2024-02-10T13:08:10.020 INFO:scrape:Failure: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh' 2024-02-10T13:08:10.020 INFO:scrape:1 jobs: ['7554381'] 2024-02-10T13:08:10.020 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-02-10T13:08:10.020 INFO:scrape: 2024-02-10T13:08:10.020 INFO:scrape:Failure: HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/25d74df18f519385e981941bd8ac52b1b1c4fd53/centos/9/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) 2024-02-10T13:08:10.020 INFO:scrape:1 jobs: ['7554448'] 2024-02-10T13:08:10.020 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-02-10T13:08:10.020 INFO:scrape: 2024-02-10T13:08:10.020 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 2024-02-10T13:08:10.020 INFO:scrape:1 jobs: ['7554721'] 2024-02-10T13:08:10.021 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-02-10T13:08:10.021 INFO:scrape: 2024-02-10T13:08:10.034 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/ failed: 49 dead: 10 running: 0 waiting: 0 queued: 0 passed: 266 Fail ================================================================= [7554331] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:35 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554331/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554331/ "2024-02-09T21:49:14.137846+0000 mon.smithi104 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554355] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:32:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554355/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554355/ "2024-02-09T22:16:13.841964+0000 mon.a (mon.0) 187 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554365] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554365/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554365/ "2024-02-09T22:05:42.030017+0000 mon.smithi112 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554369] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:51:01 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554369/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554369/ reached maximum tries (51) after waiting for 300 seconds [7554377] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:15:52 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554377/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554377/ "2024-02-09T22:12:16.463081+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554381] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 00:17:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d1ed95a945c402e858ce17e644fe5ba Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery- scrub.sh' [7554388] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:17:41 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554388/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5046e152733d487297c9b508e58c3c7d Command failed (workunit test rados/test.sh) on smithi062 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7554408] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:21:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554408/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554408/ "2024-02-09T22:34:48.118492+0000 mon.smithi084 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554418] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554418/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48c98168c611479fb9dcca021a033b05 Command failed on smithi038 with status 1: 'sudo yum install -y kernel' [7554426] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554426/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ded3f6146614d558894b91bdac01194 Failed to reconnect to smithi038 [7554432] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 01:19:06 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554432/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554432/ "2024-02-09T23:37:13.509301+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554448] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:06:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554448/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554448/ sentry: https://sentry.ceph.com/organizations/ceph/?query=532a501ff68c4481909c6b930fe7557b HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/25d74df18f519385e981941bd8ac52b1b1c4fd5 3/centos/9/flavors/default/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) [7554456] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:18 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554456/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554456/ "2024-02-09T23:03:08.555929+0000 mon.smithi018 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554464] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554464/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554464/ "2024-02-09T23:00:00.000188+0000 mon.a (mon.0) 674 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7554487] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:13 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554487/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d081431dfb44828a926c08754f2ba2d Command failed (workunit test cephadm/test_repos.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7554507] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554507/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554507/ "2024-02-09T23:24:51.978440+0000 mon.smithi043 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554517] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:38:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554517/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554517/ "2024-02-10T00:03:21.422908+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554533] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:23:18 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554533/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554533/ "2024-02-09T23:47:00.020510+0000 mon.a (mon.0) 196 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554552] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:19:32 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554552/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554552/ "2024-02-09T23:54:42.354550+0000 mon.smithi096 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554578] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:18:36 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554578/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554578/ "2024-02-10T00:05:57.258427+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554590] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:36 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554590/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554590/ "2024-02-10T00:12:45.833781+0000 mon.smithi076 (mon.0) 268 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554594] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:25:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554594/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554594/ "2024-02-10T00:19:03.229390+0000 mon.a (mon.0) 197 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554610] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:24:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554610/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554610/ "2024-02-10T00:25:57.161774+0000 mon.a (mon.0) 178 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7554629] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:50 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554629/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c09461ad5e347d1aaa80c9a925be9f0 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7554637] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:31 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554637/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554637/ "2024-02-10T00:34:34.326729+0000 mon.smithi078 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554642] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:18:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554642/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554642/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7554643] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:37:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554643/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554643/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7554649] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:22:10 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554649/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554649/ "2024-02-10T00:50:33.141104+0000 mon.a (mon.0) 200 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554651] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:35:01 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554651/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554651/ "2024-02-10T00:47:05.792502+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554652] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:18:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554652/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554652/ "2024-02-10T00:47:19.107623+0000 mon.smithi076 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554665] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:15:04 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554665/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554665/ "2024-02-10T00:41:47.994205+0000 mon.a (mon.0) 212 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7554672] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554672/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554672/ "2024-02-10T00:49:02.822507+0000 mon.smithi003 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554695] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:14:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554695/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554695/ "2024-02-10T01:00:48.447575+0000 mon.smithi037 (mon.0) 266 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554721] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 02:03:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554721/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554721/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05392666d4b04443aa3d0f3e774d5296 Command failed (workunit test rbd/test_librbd_python.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' [7554745] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:13:59 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554745/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2895dda5e5184f4cb59c7fbc4826074c Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7554751] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:45:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554751/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554751/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7554757] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:28 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554757/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554757/ "2024-02-10T01:33:46.773649+0000 mon.smithi084 (mon.0) 255 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554779] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:15:35 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554779/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554779/ "2024-02-10T01:44:18.982999+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554787] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:40:20 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554787/ "2024-02-10T02:10:00.000208+0000 mon.a (mon.0) 1750 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7554795] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554795/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554795/ "2024-02-10T01:57:55.029689+0000 mon.smithi155 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554817] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:29:29 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6dcdad5f9b5846a78fcaf4793a9c3878 "2024-02-10T02:30:00.000220+0000 mon.a (mon.0) 1284 : cluster 3 [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline" in cluster log [7554829] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:25:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554829/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554829/ "2024-02-10T02:40:26.221070+0000 mon.a (mon.0) 952 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7554835] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:12:53 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554835/ "2024-02-10T02:27:10.022384+0000 mon.smithi037 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7554866] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:25:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554866/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554866/ "2024-02-10T02:54:31.472240+0000 mon.a (mon.0) 387 : cluster 3 [WRN] MON_DOWN: 1/5 mons down, quorum a,e,c,d" in cluster log [7554868] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:17:11 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554868/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554868/ "2024-02-10T02:57:29.780920+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7554892] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554892/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554892/ "2024-02-10T03:20:00.000254+0000 mon.a (mon.0) 849 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7554898] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:10:13 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554898/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554898/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7554904] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:54 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554904/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b54947c2ddf040a582b475175555ad91 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7554912] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:20:07 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554912/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554912/ "2024-02-10T03:38:48.715064+0000 mon.a (mon.0) 196 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log Dead ================================================================= [7554343] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554343/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554343/ Error reimaging machines: Failed to power on smithi070 [7554410] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554410/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554410/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7554444] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554444/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554444/ hit max job timeout [7554519] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554519/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554519/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7554548] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554548/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554548/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7554586] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554586/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554586/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7554655] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554655/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554655/ Error reimaging machines: Failed to power on smithi026 [7554717] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554717/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554717/ hit max job timeout [7554841] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554841/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554841/ Error reimaging machines: Failed to power on smithi150 [7554876] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554876/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554876/ Error reimaging machines: Failed to power on smithi063 Pass ================================================================= [7554333] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:09:48 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554333/ [7554335] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:17:15 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554335/ [7554337] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:12:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554337/ [7554339] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:02 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554339/ [7554341] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:09 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554341/ [7554345] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554345/ [7554347] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:43 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554347/ [7554349] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} time: 00:07:26 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554349/ [7554351] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:22:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554351/ [7554353] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:15:07 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554353/ [7554357] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554357/ [7554359] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} time: 00:15:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554359/ [7554361] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:59 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554361/ [7554363] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:20:04 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554363/ [7554367] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:17:06 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554367/ [7554371] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554371/ [7554373] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:23:40 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554373/ [7554375] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554375/ [7554379] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554379/ [7554383] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554383/ [7554385] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:36:28 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554385/ [7554386] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:25:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554386/ [7554390] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:24:45 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554390/ [7554392] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554392/ [7554394] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:30:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554394/ [7554396] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:07:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554396/ [7554398] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} time: 00:11:51 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554398/ [7554400] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:31:13 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554400/ [7554402] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:07:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554402/ [7554404] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554404/ [7554406] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554406/ [7554412] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:17:28 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554412/ [7554414] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:14:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554414/ [7554416] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:45:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554416/ [7554420] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:02 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554420/ [7554422] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:32 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554422/ [7554424] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:30:29 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554424/ [7554428] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554428/ [7554430] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:29:21 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554430/ [7554434] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554434/ [7554436] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:28:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554436/ [7554438] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:45 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554438/ [7554440] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:40:01 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554440/ [7554442] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:12:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554442/ [7554446] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:10 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554446/ [7554450] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:52 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554450/ [7554452] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:12:04 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554452/ [7554454] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:17:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554454/ [7554458] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:18:12 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554458/ [7554460] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:07:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554460/ [7554462] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:13:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554462/ [7554466] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554466/ [7554468] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:56 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554468/ [7554470] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} time: 00:33:10 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554470/ [7554472] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554472/ [7554474] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:21:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554474/ [7554476] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} time: 00:11:20 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554476/ [7554478] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:16:22 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554478/ [7554479] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:53 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554479/ [7554481] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:41 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554481/ [7554483] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554483/ [7554485] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:26:48 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554485/ [7554489] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:13:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554489/ [7554491] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:22:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554491/ [7554493] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:04 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554493/ [7554495] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} time: 00:13:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554495/ [7554497] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:27:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554497/ [7554499] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:19:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554499/ [7554501] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:10:40 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554501/ [7554503] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:56 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554503/ [7554505] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:15:32 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554505/ [7554509] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:20:39 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554509/ [7554511] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554511/ [7554513] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554513/ [7554515] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:21:52 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554515/ [7554521] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:14:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554521/ [7554523] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/rados_python} time: 00:16:21 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554523/ [7554525] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:12:28 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554525/ [7554527] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:39 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554527/ [7554529] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:54 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554529/ [7554531] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:24:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554531/ [7554535] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:15:57 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554535/ [7554537] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:09:44 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554537/ [7554539] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:17:40 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554539/ [7554541] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554541/ [7554543] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:03 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554543/ [7554545] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554545/ [7554547] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:22:35 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554547/ [7554550] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:13:26 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554550/ [7554554] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554554/ [7554556] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:33:15 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554556/ [7554558] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:52 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554558/ [7554560] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:15 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554560/ [7554562] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:14:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554562/ [7554564] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554564/ [7554566] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:24:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554566/ [7554568] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:19:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554568/ [7554570] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:18 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554570/ [7554572] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:41 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554572/ [7554574] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:30:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554574/ [7554576] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:58 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554576/ [7554580] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554580/ [7554582] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:33:31 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554582/ [7554584] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:33:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554584/ [7554588] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:19:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554588/ [7554592] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554592/ [7554596] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:17 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554596/ [7554598] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554598/ [7554600] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:21:29 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554600/ [7554602] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554602/ [7554604] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:51 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554604/ [7554606] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/e2e} time: 00:23:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554606/ [7554608] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} time: 00:10:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554608/ [7554612] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:18:57 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554612/ [7554614] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554614/ [7554616] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:11:45 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554616/ [7554618] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:08:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554618/ [7554620] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:07:44 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554620/ [7554622] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:40 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554622/ [7554624] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554624/ [7554626] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:23:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554626/ [7554627] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:21:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554627/ [7554628] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:15:50 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554628/ [7554630] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli_mon} time: 00:24:09 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554630/ [7554631] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} time: 00:14:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554631/ [7554632] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:18:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554632/ [7554633] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:31:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554633/ [7554634] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:20:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554634/ [7554635] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554635/ [7554636] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:50 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554636/ [7554638] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:59 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554638/ [7554639] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:17:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554639/ [7554640] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:19:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554640/ [7554641] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:15:37 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554641/ [7554644] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:36:51 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554644/ [7554645] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554645/ [7554646] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554646/ [7554647] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_big} time: 00:29:07 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554647/ [7554648] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:30:12 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554648/ [7554650] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554650/ [7554653] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554653/ [7554654] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:41:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554654/ [7554656] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:11:11 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554656/ [7554657] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:23:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554657/ [7554658] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:20:41 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554658/ [7554659] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:13 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554659/ [7554660] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554660/ [7554661] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_adoption} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554661/ [7554662] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} time: 00:13:17 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554662/ [7554663] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:26 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554663/ [7554664] rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} time: 01:05:06 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554664/ [7554666] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:33:20 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554666/ [7554667] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} time: 00:13:52 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554667/ [7554668] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:21:45 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554668/ [7554669] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:11 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554669/ [7554670] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:41 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554670/ [7554671] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} time: 00:19:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554671/ [7554673] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:39:17 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554673/ [7554674] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:12:23 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554674/ [7554675] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:34 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554675/ [7554676] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554676/ [7554677] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:26:29 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554677/ [7554678] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:58 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554678/ [7554679] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554679/ [7554681] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554681/ [7554683] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:16:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554683/ [7554685] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} time: 00:19:22 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554685/ [7554687] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:29:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554687/ [7554689] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:10 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554689/ [7554691] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:22:03 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554691/ [7554693] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:40 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554693/ [7554697] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554697/ [7554699] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:55:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554699/ [7554701] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:58:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554701/ [7554703] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:23:53 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554703/ [7554705] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554705/ [7554707] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:34 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554707/ [7554709] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:16:53 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554709/ [7554711] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:21 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554711/ [7554713] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:24:13 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554713/ [7554715] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} time: 01:10:59 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554715/ [7554719] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:26:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554719/ [7554723] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 00:59:20 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554723/ [7554725] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:09:56 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554725/ [7554727] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554727/ [7554729] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:13:06 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554729/ [7554731] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:20:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554731/ [7554733] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554733/ [7554735] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:22:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554735/ [7554737] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554737/ [7554739] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554739/ [7554741] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:08:52 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554741/ [7554743] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554743/ [7554747] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554747/ [7554749] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:03 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554749/ [7554753] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:13:30 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554753/ [7554755] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 02:12:21 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554755/ [7554759] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:15:19 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554759/ [7554761] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:23:54 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554761/ [7554763] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:15:43 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554763/ [7554765] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:06 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554765/ [7554767] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:19:31 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554767/ [7554769] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554769/ [7554771] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:14:47 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554771/ [7554773] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:34:44 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554773/ [7554775] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} time: 00:31:11 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554775/ [7554777] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} time: 01:01:45 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554777/ [7554781] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554781/ [7554783] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:07:56 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554783/ [7554785] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:22 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554785/ [7554789] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:23:01 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554789/ [7554791] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:21:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554791/ [7554793] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:18:57 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554793/ [7554797] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554797/ [7554799] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:20:57 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554799/ [7554801] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:39 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554801/ [7554803] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:22:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554803/ [7554805] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:12:58 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554805/ [7554807] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:18:58 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554807/ [7554809] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/repair_test} time: 00:15:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554809/ [7554811] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554811/ [7554813] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554813/ [7554815] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:09:01 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554815/ [7554819] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:54 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554819/ [7554821] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554821/ [7554823] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:19:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554823/ [7554825] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:11:58 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554825/ [7554827] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:07:54 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554827/ [7554831] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:25:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554831/ [7554833] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:53 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554833/ [7554837] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:01 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554837/ [7554839] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554839/ [7554843] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:31:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554843/ [7554845] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:18:38 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554845/ [7554848] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554848/ [7554850] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli} time: 00:12:57 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554850/ [7554852] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:09:02 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554852/ [7554854] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:50 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554854/ [7554856] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554856/ [7554858] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:27:27 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554858/ [7554860] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:19:07 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554860/ [7554862] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554862/ [7554864] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/scrub_test} time: 00:09:49 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554864/ [7554870] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:31 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554870/ [7554872] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:25 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554872/ [7554874] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:09:55 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554874/ [7554878] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:22 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554878/ [7554880] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:19:33 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554880/ [7554882] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/e2e} time: 00:22:48 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554882/ [7554884] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554884/ [7554886] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:26:24 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554886/ [7554888] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:22:08 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554888/ [7554890] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:59 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554890/ [7554894] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:12:28 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554894/ [7554896] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:16 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554896/ [7554900] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554900/ [7554902] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:45:31 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554902/ [7554906] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:20 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554906/ [7554908] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554908/ [7554910] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:31:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554910/ [7554914] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:16:32 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554914/ [7554916] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:13:44 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554916/ [7554918] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:42 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554918/ [7554920] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:26 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554920/ [7554922] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:16:39 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554922/ [7554924] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554924/