2024-03-01T22:33:53.851 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-01T22:33:53.851 INFO:teuthology.results:subset: '111/120000' 2024-03-01T22:33:53.852 INFO:teuthology.results:seed: '6722' 2024-03-02T06:33:51.276 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-02T06:33:51.324 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-02T16:57:48.775 INFO:teuthology.results:Tests finished! gathering results... 2024-03-02T16:58:18.431 INFO:scrape:Found 325 jobs 2024-03-02T16:58:20.368 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.369 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.369 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.369 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.370 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.370 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.370 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.370 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.371 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.371 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.371 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:20.371 WARNING:scrape:Saw end of BT but not start 2024-03-02T16:58:26.881 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578582/teuthology.log 2024-03-02T16:58:27.542 INFO:scrape:Found 10 distinct failure reasons 2024-03-02T16:58:27.543 INFO:scrape:Failure: "2024-03-02T02:00:00.000218+0000 mon.a (mon.0) 655 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log 2024-03-02T16:58:27.543 INFO:scrape:19 jobs: ['7578332', '7578047', '7578205', '7578609', '7578549', '7578145', '7578418', '7578505', '7578217', '7578314', '7578688', '7578653', '7578729', '7578293', '7578710', '7578805', '7578766', '7578362', '7578081'] 2024-03-02T16:58:27.543 INFO:scrape:suites intersection: [] 2024-03-02T16:58:27.543 INFO:scrape:suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-03-02T16:58:27.543 INFO:scrape: 2024-03-02T16:58:27.543 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-03-02T16:58:27.543 INFO:scrape:2 jobs: ['7578481', '7578834'] 2024-03-02T16:58:27.544 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output'] 2024-03-02T16:58:27.544 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-02T16:58:27.544 INFO:scrape: 2024-03-02T16:58:27.544 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-02T16:58:27.544 INFO:scrape:5 jobs: ['7578828', '7578520', '7578086', '7578707', '7578311'] 2024-03-02T16:58:27.544 INFO:scrape:suites intersection: [] 2024-03-02T16:58:27.545 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-03-02T16:58:27.545 INFO:scrape: 2024-03-02T16:58:27.545 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2024-03-02T16:58:27.545 INFO:scrape:1 jobs: ['7578704'] 2024-03-02T16:58:27.545 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_cephadm_timeout}'] 2024-03-02T16:58:27.545 INFO:scrape: 2024-03-02T16:58:27.545 INFO:scrape:Failure: SSH connection to smithi184 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' 2024-03-02T16:58:27.545 INFO:scrape:1 jobs: ['7578642'] 2024-03-02T16:58:27.545 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-03-02T16:58:27.545 INFO:scrape: 2024-03-02T16:58:27.546 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-02T16:58:27.546 INFO:scrape:1 jobs: ['7578104'] 2024-03-02T16:58:27.546 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}'] 2024-03-02T16:58:27.546 INFO:scrape: 2024-03-02T16:58:27.546 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-03-02T16:58:27.546 INFO:scrape:1 jobs: ['7578263'] 2024-03-02T16:58:27.546 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-03-02T16:58:27.547 INFO:scrape: 2024-03-02T16:58:27.547 INFO:scrape:Dead: b'2024-03-02T12:05:21.752 DEBUG:teuthology.exit:Finished running handlers' 2024-03-02T16:58:27.547 INFO:scrape:2 jobs: ['7578199', '7578684'] 2024-03-02T16:58:27.547 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}'] 2024-03-02T16:58:27.547 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}'] 2024-03-02T16:58:27.547 INFO:scrape: 2024-03-02T16:58:27.547 INFO:scrape:Dead: None 2024-03-02T16:58:27.547 INFO:scrape:5 jobs: ['7578582', '7578266', '7578665', '7578400', '7578523'] 2024-03-02T16:58:27.547 INFO:scrape: 2024-03-02T16:58:27.547 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-03-02T16:58:27.548 INFO:scrape:1 jobs: ['7578115'] 2024-03-02T16:58:27.548 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-03-02T16:58:27.548 INFO:scrape: 2024-03-02T16:58:27.560 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/ failed: 30 dead: 8 running: 0 waiting: 0 queued: 0 passed: 287 Fail ================================================================= [7578047] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:34 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578047/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578047/ "2024-03-01T22:52:11.205846+0000 mon.smithi016 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578081] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578081/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578081/ "2024-03-01T23:09:29.681691+0000 mon.smithi047 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578086] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:51:39 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578086/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578086/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7578104] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:49:55 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578104/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578104/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a4010534db148c5bc8b577a95a900f2 Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7578115] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:17:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578115/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578115/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef9c5ac1ba2c4d42a2349025b22bdeba Command failed (workunit test rados/test.sh) on smithi082 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7578145] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:21:08 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578145/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578145/ "2024-03-01T23:48:27.727710+0000 mon.smithi071 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578205] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:46:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578205/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578205/ "2024-03-02T00:29:38.722060+0000 mon.a (mon.0) 102 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7578217] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:02 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578217/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578217/ "2024-03-02T00:37:04.065990+0000 mon.smithi032 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578263] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:24 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578263/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578263/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85b612fd5826486084abb09060b8e9e5 Command failed (workunit test cephadm/test_repos.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7578293] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:14:10 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578293/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578293/ "2024-03-02T01:16:02.220834+0000 mon.smithi079 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578311] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:57:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578311/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578311/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7578314] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:13:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578314/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578314/ "2024-03-02T01:37:21.685647+0000 mon.a (mon.0) 8 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log [7578332] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:23:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578332/ "2024-03-02T02:00:00.000218+0000 mon.a (mon.0) 655 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7578362] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:20:02 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578362/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578362/ "2024-03-02T02:08:57.439039+0000 mon.smithi023 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578418] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:15:08 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578418/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578418/ "2024-03-02T02:39:08.395333+0000 mon.smithi023 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578481] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578481/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578481/ sentry: https://sentry.ceph.com/organizations/ceph/?query=485147246e4e43a4aec834739764ee43 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7578505] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:16:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578505/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578505/ "2024-03-02T03:26:40.760420+0000 mon.smithi006 (mon.0) 286 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578520] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:26:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578520/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578520/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7578549] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:18:55 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578549/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578549/ "2024-03-02T03:56:01.865546+0000 mon.smithi079 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578609] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578609/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578609/ "2024-03-02T04:30:28.462700+0000 mon.smithi045 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578653] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:14:08 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578653/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578653/ "2024-03-02T04:48:38.509819+0000 mon.smithi078 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578688] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 02:03:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578688/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578688/ "2024-03-02T05:10:57.057062+0000 mon.a (mon.0) 183 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7578704] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:14:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578704/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4505b08318e24598874c6abdedc7bc34 Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c76c50a73f63ba48ccdf0adccce03b00d1d80c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7578707] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:46:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578707/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578707/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7578710] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:18 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578710/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578710/ "2024-03-02T05:23:26.622346+0000 mon.smithi016 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578729] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578729/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578729/ "2024-03-02T05:30:10.941144+0000 mon.smithi105 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578766] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:15:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578766/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578766/ "2024-03-02T05:54:45.633794+0000 mon.smithi145 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578805] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:37 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578805/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578805/ "2024-03-02T06:14:23.686888+0000 mon.smithi063 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7578828] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:04:58 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578828/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578828/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7578834] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:11:41 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8abbdb8537344168e41d16bc2e72f94 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' Dead ================================================================= [7578199] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578199/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578199/ hit max job timeout [7578266] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578266/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578266/ Error reimaging machines: Failed to power on smithi106 [7578400] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578400/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578400/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7578523] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578523/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578523/ Error reimaging machines: Expected smithi077's OS to be centos 9 but found ubuntu 22.04 [7578582] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578582/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578582/ Error reimaging machines: Expected smithi131's OS to be ubuntu 22.04 but found centos 9 [7578642] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578642/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578642/ sentry: https://sentry.ceph.com/organizations/ceph/?query=663464964d3f444896fee7fbf0be8cc2 SSH connection to smithi184 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' [7578665] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578665/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578665/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7578684] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578684/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578684/ hit max job timeout Pass ================================================================= [7578049] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:12:35 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578049/ [7578051] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578051/ [7578053] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578053/ [7578055] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578055/ [7578057] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578057/ [7578059] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:38:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578059/ [7578061] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} time: 00:15:44 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578061/ [7578063] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:37 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578063/ [7578065] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:09:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578065/ [7578067] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:22:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578067/ [7578069] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578069/ [7578071] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:25:18 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578071/ [7578073] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:26 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578073/ [7578075] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} time: 00:14:36 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578075/ [7578077] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:41 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578077/ [7578079] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:15:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578079/ [7578083] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:12:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578083/ [7578089] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578089/ [7578092] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:24:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578092/ [7578095] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578095/ [7578098] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:16:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578098/ [7578101] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578101/ [7578106] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578106/ [7578109] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:39:34 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578109/ [7578113] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:26:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578113/ [7578118] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:25:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578118/ [7578121] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:40 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578121/ [7578124] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_workunits} time: 00:30:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578124/ [7578127] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:09:39 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578127/ [7578130] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} time: 00:12:24 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578130/ [7578133] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:48:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578133/ [7578136] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:07:49 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578136/ [7578139] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:40 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578139/ [7578143] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:22:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578143/ [7578149] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:28:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578149/ [7578152] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:19:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578152/ [7578154] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578154/ [7578157] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:42:45 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578157/ [7578161] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578161/ [7578163] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:35 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578163/ [7578166] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:57 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578166/ [7578169] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:30:36 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578169/ [7578172] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:15:50 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578172/ [7578175] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:52 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578175/ [7578178] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:30:06 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578178/ [7578181] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:42:41 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578181/ [7578184] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} time: 00:14:51 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578184/ [7578187] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/dashboard} time: 01:11:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578187/ [7578190] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/crash} time: 00:13:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578190/ [7578193] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:40:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578193/ [7578196] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:14:57 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578196/ [7578202] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578202/ [7578208] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:28 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578208/ [7578211] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:13:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578211/ [7578214] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:16:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578214/ [7578220] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:41 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578220/ [7578223] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:09:23 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578223/ [7578226] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:13:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578226/ [7578229] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578229/ [7578231] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578231/ [7578234] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578234/ [7578237] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} time: 00:33:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578237/ [7578239] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:13:23 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578239/ [7578242] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:21:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578242/ [7578245] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578245/ [7578248] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:17:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578248/ [7578251] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:08:44 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578251/ [7578254] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:07 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578254/ [7578257] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:30 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578257/ [7578260] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:27:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578260/ [7578269] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:22:29 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578269/ [7578272] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:30 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578272/ [7578275] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578275/ [7578278] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:28:41 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578278/ [7578281] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:20:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578281/ [7578284] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:58 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578284/ [7578287] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578287/ [7578290] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:15:36 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578290/ [7578296] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:23:48 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578296/ [7578299] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578299/ [7578302] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:32 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578302/ [7578305] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:22:30 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578305/ [7578308] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:23:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578308/ [7578317] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/rados_python} time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578317/ [7578320] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:11:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578320/ [7578323] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:55 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578323/ [7578326] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578326/ [7578329] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:24:30 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578329/ [7578336] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:16:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578336/ [7578339] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:09:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578339/ [7578342] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:17:28 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578342/ [7578345] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578345/ [7578348] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:22:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578348/ [7578351] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:12:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578351/ [7578354] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:24:48 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578354/ [7578356] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:29 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578356/ [7578359] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:15:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578359/ [7578365] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578365/ [7578367] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:28:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578367/ [7578370] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578370/ [7578373] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:52 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578373/ [7578376] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:36 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578376/ [7578379] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:21:06 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578379/ [7578382] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:27:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578382/ [7578385] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:17:37 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578385/ [7578388] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:45 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578388/ [7578391] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:18 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578391/ [7578394] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} time: 01:21:08 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578394/ [7578397] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578397/ [7578403] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:35 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578403/ [7578406] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:33:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578406/ [7578409] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:21:32 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578409/ [7578412] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:21:23 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578412/ [7578415] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:17:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578415/ [7578421] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:29 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578421/ [7578424] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:26:26 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578424/ [7578427] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578427/ [7578430] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:18 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578430/ [7578433] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:17:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578433/ [7578436] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578436/ [7578439] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:17 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578439/ [7578442] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/e2e} time: 00:25:59 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578442/ [7578445] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} time: 00:16:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578445/ [7578448] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:25:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578448/ [7578451] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:16:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578451/ [7578454] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:07 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578454/ [7578457] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:07:50 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578457/ [7578460] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:30 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578460/ [7578463] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:09:39 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578463/ [7578466] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:24:01 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578466/ [7578469] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} time: 00:12:38 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578469/ [7578472] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:25:52 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578472/ [7578475] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:18:38 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578475/ [7578478] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} time: 00:19:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578478/ [7578484] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli_mon} time: 00:32:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578484/ [7578487] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} time: 00:16:52 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578487/ [7578490] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:18:20 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578490/ [7578493] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:54:41 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578493/ [7578496] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:21:20 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578496/ [7578499] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578499/ [7578502] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:28 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578502/ [7578508] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:56 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578508/ [7578511] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:19:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578511/ [7578514] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:25:34 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578514/ [7578517] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:14:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578517/ [7578526] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:32:26 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578526/ [7578529] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578529/ [7578532] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:06 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578532/ [7578535] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_big} time: 00:30:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578535/ [7578538] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:27:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578538/ [7578541] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:22:24 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578541/ [7578544] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:56 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578544/ [7578547] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:35:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578547/ [7578552] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:20:10 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578552/ [7578555] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:30:17 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578555/ [7578558] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} time: 00:16:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578558/ [7578561] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats} time: 00:16:10 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578561/ [7578564] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:49 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578564/ [7578567] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:17:50 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578567/ [7578570] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:08:05 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578570/ [7578573] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:16:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578573/ [7578576] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_adoption} time: 00:14:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578576/ [7578579] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} time: 00:13:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578579/ [7578585] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:07:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578585/ [7578588] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:23 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578588/ [7578591] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:32:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578591/ [7578594] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} time: 00:12:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578594/ [7578597] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578597/ [7578600] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:13 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578600/ [7578603] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:32 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578603/ [7578606] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:22:57 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578606/ [7578612] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:56 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578612/ [7578615] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:56 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578615/ [7578618] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578618/ [7578621] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:12:07 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578621/ [7578624] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:29:35 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578624/ [7578627] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:52 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578627/ [7578630] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:18 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578630/ [7578633] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:19 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578633/ [7578636] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:18:16 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578636/ [7578639] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} time: 00:19:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578639/ [7578645] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:31:10 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578645/ [7578647] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/progress} time: 00:20:59 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578647/ [7578650] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:51 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578650/ [7578656] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578656/ [7578659] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:58:55 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578659/ [7578662] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:57:34 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578662/ [7578668] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578668/ [7578670] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:05:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578670/ [7578674] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:17:19 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578674/ [7578677] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578677/ [7578680] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:24:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578680/ [7578682] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:09:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578682/ [7578686] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} time: 00:25:28 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578686/ [7578690] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:02:58 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578690/ [7578692] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578692/ [7578694] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:34 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578694/ [7578696] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578696/ [7578697] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:19:50 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578697/ [7578698] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578698/ [7578699] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:22:35 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578699/ [7578700] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:16 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578700/ [7578701] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:27 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578701/ [7578702] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:10:45 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578702/ [7578703] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:13:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578703/ [7578705] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578705/ [7578706] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:18:40 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578706/ [7578708] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578708/ [7578709] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 02:13:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578709/ [7578711] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578711/ [7578712] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:29:44 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578712/ [7578713] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:15:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578713/ [7578714] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578714/ [7578715] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:19:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578715/ [7578716] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:33 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578716/ [7578717] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:17:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578717/ [7578718] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:37:48 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578718/ [7578719] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} time: 00:29:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578719/ [7578720] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} time: 01:02:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578720/ [7578721] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} time: 00:18:45 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578721/ [7578722] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:34 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578722/ [7578723] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:08 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578723/ [7578724] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578724/ [7578725] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:34:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578725/ [7578726] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:20:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578726/ [7578727] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:25:44 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578727/ [7578728] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:23 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578728/ [7578730] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:08 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578730/ [7578731] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:20:24 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578731/ [7578732] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:57 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578732/ [7578734] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:22:14 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578734/ [7578736] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578736/ [7578738] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:21:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578738/ [7578740] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:18:29 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578740/ [7578742] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:21:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578742/ [7578744] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578744/ [7578746] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:15:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578746/ [7578748] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:38 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578748/ [7578750] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578750/ [7578752] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578752/ [7578754] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:22:30 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578754/ [7578756] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:12:28 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578756/ [7578758] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:08:06 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578758/ [7578760] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:25:17 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578760/ [7578762] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578762/ [7578764] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:02 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578764/ [7578768] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:10 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578768/ [7578770] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:42 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578770/ [7578772] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:38:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578772/ [7578774] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:31:02 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578774/ [7578776] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:23:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578776/ [7578778] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578778/ [7578780] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli} time: 00:13:01 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578780/ [7578781] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578781/ [7578783] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:59 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578783/ [7578785] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578785/ [7578787] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:28:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578787/ [7578789] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:21:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578789/ [7578791] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:12:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578791/ [7578793] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:28 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578793/ [7578795] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:34:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578795/ [7578797] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:25:20 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578797/ [7578799] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578799/ [7578801] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:57 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578801/ [7578803] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:14:32 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578803/ [7578807] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:18 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578807/ [7578809] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:22:06 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578809/ [7578811] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/e2e} time: 00:29:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578811/ [7578814] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578814/ [7578816] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:23:29 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578816/ [7578818] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:19:19 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578818/ [7578820] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:43 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578820/ [7578822] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:17:19 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578822/ [7578824] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:12:23 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578824/ [7578826] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578826/ [7578830] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578830/ [7578832] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:42:26 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578832/ [7578836] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:19 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578836/ [7578838] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578838/ [7578840] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 02:46:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578840/ [7578842] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_rgw_multisite} time: 00:19:11 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578842/ [7578844] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:52 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578844/ [7578846] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578846/ [7578848] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578848/ [7578850] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578850/ [7578852] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:16:00 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578852/ [7578854] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:11:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:50:47-rados-squid-distro-default-smithi/7578854/