2024-03-22T08:27:06.259 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-22T08:27:06.259 INFO:teuthology.results:subset: '30391/100000' 2024-03-22T08:27:06.259 INFO:teuthology.results:seed: '7188' 2024-03-25T09:52:28.362 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-25T09:52:28.385 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-25T21:52:44.787 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-03-25T21:53:11.475 INFO:scrape:Found 306 jobs 2024-03-25T21:53:14.851 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615311/teuthology.log 2024-03-25T21:53:22.291 INFO:scrape:Found 12 distinct failure reasons 2024-03-25T21:53:22.292 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-03-25T21:53:22.292 INFO:scrape:2 jobs: ['7615423', '7615271'] 2024-03-25T21:53:22.292 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-25T21:53:22.292 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-25T21:53:22.292 INFO:scrape: 2024-03-25T21:53:22.292 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=768f6cd6e95d0c7a626042719c230df59bdba9d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-25T21:53:22.292 INFO:scrape:1 jobs: ['7615457'] 2024-03-25T21:53:22.293 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-03-25T21:53:22.293 INFO:scrape: 2024-03-25T21:53:22.293 INFO:scrape:Failure: ["E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem."] 2024-03-25T21:53:22.293 INFO:scrape:1 jobs: ['7615361'] 2024-03-25T21:53:22.293 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-radosbench}'] 2024-03-25T21:53:22.293 INFO:scrape: 2024-03-25T21:53:22.293 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T21:53:22.293 INFO:scrape:4 jobs: ['7615379', '7615210', '7615446', '7615283'] 2024-03-25T21:53:22.293 INFO:scrape:suites intersection: [] 2024-03-25T21:53:22.293 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] 2024-03-25T21:53:22.293 INFO:scrape: 2024-03-25T21:53:22.294 INFO:scrape:Failure: "2024-03-25T09:33:15.797682+0000 mon.smithi062 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log 2024-03-25T21:53:22.294 INFO:scrape:17 jobs: ['7615439', '7615314', '7615395', '7615213', '7615435', '7615373', '7615338', '7615355', '7615249', '7615414', '7615231', '7615328', '7615492', '7615273', '7615296', '7615475', '7615494'] 2024-03-25T21:53:22.294 INFO:scrape:suites intersection: [] 2024-03-25T21:53:22.294 INFO:scrape:suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'task/test_extra_daemon_features}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-03-25T21:53:22.294 INFO:scrape: 2024-03-25T21:53:22.294 INFO:scrape:Dead: b'2024-03-25T09:14:58.612 DEBUG:teuthology.exit:Finished running handlers' 2024-03-25T21:53:22.294 INFO:scrape:3 jobs: ['7615336', '7615455', '7615490'] 2024-03-25T21:53:22.294 INFO:scrape:suites intersection: [] 2024-03-25T21:53:22.295 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-25T21:53:22.295 INFO:scrape: 2024-03-25T21:53:22.295 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=768f6cd6e95d0c7a626042719c230df59bdba9d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-03-25T21:53:22.295 INFO:scrape:1 jobs: ['7615404'] 2024-03-25T21:53:22.295 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-03-25T21:53:22.295 INFO:scrape: 2024-03-25T21:53:22.295 INFO:scrape:Dead: None 2024-03-25T21:53:22.295 INFO:scrape:3 jobs: ['7615311', '7615310', '7615360'] 2024-03-25T21:53:22.295 INFO:scrape: 2024-03-25T21:53:22.295 INFO:scrape:Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2024-03-25T21:53:22.295 INFO:scrape:1 jobs: ['7615470'] 2024-03-25T21:53:22.296 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_orch_cli}'] 2024-03-25T21:53:22.296 INFO:scrape: 2024-03-25T21:53:22.296 INFO:scrape:Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6593 (apt-get)'] 2024-03-25T21:53:22.296 INFO:scrape:1 jobs: ['7615229'] 2024-03-25T21:53:22.296 INFO:scrape:suites: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/connectivity', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/crash}'] 2024-03-25T21:53:22.296 INFO:scrape: 2024-03-25T21:53:22.296 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=768f6cd6e95d0c7a626042719c230df59bdba9d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2024-03-25T21:53:22.296 INFO:scrape:1 jobs: ['7615284'] 2024-03-25T21:53:22.296 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_timeout}'] 2024-03-25T21:53:22.296 INFO:scrape: 2024-03-25T21:53:22.297 INFO:scrape:Failure: Command failed on smithi105 with status 5: 'sudo systemctl stop ceph-8890c5e8-e82d-11ee-95cd-87774f69a715@mon.a' 2024-03-25T21:53:22.297 INFO:scrape:1 jobs: ['7615245'] 2024-03-25T21:53:22.297 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-03-25T21:53:22.297 INFO:scrape: 2024-03-25T21:53:22.310 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/ failed: 28 dead: 7 running: 1 waiting: 0 queued: 0 passed: 270 Fail ================================================================= [7615210] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:10:05 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615210/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615210/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7615213] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:24 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615213/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615213/ "2024-03-22T08:51:17.618105+0000 mon.smithi104 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615231] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:19:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615231/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615231/ "2024-03-22T09:08:32.958765+0000 mon.smithi007 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615245] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615245/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615245/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5f42dd57a9542708cb9908547994c69 Command failed on smithi105 with status 5: 'sudo systemctl stop ceph-8890c5e8-e82d-11ee-95cd-87774f69a715@mon.a' [7615249] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615249/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615249/ "2024-03-24T20:47:52.182085+0000 mon.smithi080 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615271] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:48 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615271/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615271/ sentry: https://sentry.ceph.com/organizations/ceph/?query=430fab7f5e144168bd763b763ba87ed4 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7615273] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:15:07 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615273/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615273/ "2024-03-24T20:58:35.624850+0000 mon.smithi045 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615283] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:19:54 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615283/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615283/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7615284] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:19:27 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615284/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8387775699f246aba5a6001025e491f0 Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=768f6cd6e95d0c7a626042719c230df59bdba9d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7615296] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:18:58 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615296/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615296/ "2024-03-24T21:16:55.821176+0000 mon.smithi121 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615314] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:54 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615314/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615314/ "2024-03-24T21:19:41.563944+0000 mon.smithi005 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615328] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:40 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615328/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615328/ "2024-03-24T21:24:28.791639+0000 mon.smithi172 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615338] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:47:30 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615338/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615338/ "2024-03-24T21:38:36.978441+0000 mon.a (mon.0) 131 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7615355] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615355/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615355/ "2024-03-24T21:42:52.359635+0000 mon.smithi067 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615373] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615373/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615373/ "2024-03-25T08:49:52.819078+0000 mon.smithi033 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615379] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:32:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615379/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615379/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7615395] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:17 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615395/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615395/ "2024-03-25T09:13:33.464721+0000 mon.smithi083 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615404] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:29 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615404/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96074d5fa1da403e80957776668bc67e Command failed (workunit test cephadm/test_repos.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=768f6cd6e95d0c7a626042719c230df59bdba9d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7615414] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:21:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615414/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615414/ "2024-03-25T09:24:40.124104+0000 mon.smithi029 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615423] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:15 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615423/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615423/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aceddfe46f5b4317b3ce4436986a129d Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7615435] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:15:38 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615435/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615435/ "2024-03-25T09:37:39.453352+0000 mon.a (mon.0) 8 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log [7615439] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615439/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615439/ "2024-03-25T09:33:15.797682+0000 mon.smithi062 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615446] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:58:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615446/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615446/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7615457] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:33:19 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615457/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=951682e32ecd42a7a93aab020f0bec78 Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=768f6cd6e95d0c7a626042719c230df59bdba9d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7615470] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:21:50 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615470/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffa0362583274f4f9dcc74cd002a02e0 Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7615475] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:20:44 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615475/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615475/ "2024-03-25T09:55:11.416340+0000 mon.smithi153 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7615492] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:49:56 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615492/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615492/ "2024-03-25T11:00:00.000145+0000 mon.a (mon.0) 3011 : cluster 3 [WRN] TELEMETRY_CHANGED: Telemetry requires re-opt-in" in cluster log [7615494] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615494/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615494/ "2024-03-25T10:00:23.366753+0000 mon.smithi059 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log Dead ================================================================= [7615229] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/crash} ----------------------------------------------------------------- time: 00:06:56 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615229/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fade1ef5555435bb97a2162aebee813 ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6593 (apt-get)'] [7615310] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615310/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615310/ Error reimaging machines: Failed to power on smithi055 [7615311] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615311/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615311/ Error reimaging machines: Failed to power on smithi073 [7615336] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615336/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615336/ hit max job timeout [7615360] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615360/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615360/ Error reimaging machines: Failed to power on smithi186 [7615361] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 00:07:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615361/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b7f81233c3d491e92fc7859a6fca934 ["E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem."] [7615455] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615455/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615455/ hit max job timeout Running ================================================================= [7615490] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615490/ Pass ================================================================= [7615206] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:45 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615206/ [7615207] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:40 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615207/ [7615208] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:45 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615208/ [7615209] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli_mon} time: 00:24:38 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615209/ [7615211] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:25:41 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615211/ [7615212] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:22:11 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615212/ [7615214] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:58 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615214/ [7615215] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:12:13 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615215/ [7615216] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:19:51 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615216/ [7615217] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:19:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615217/ [7615218] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:34:03 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615218/ [7615219] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:28 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615219/ [7615220] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:08:36 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615220/ [7615221] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:13 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615221/ [7615222] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:15:55 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615222/ [7615223] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:17 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615223/ [7615224] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:21:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615224/ [7615225] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:34 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615225/ [7615226] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:22:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615226/ [7615227] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:09:56 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615227/ [7615228] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_api_tests} time: 00:34:46 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615228/ [7615230] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:24:47 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615230/ [7615232] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 01:01:03 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615232/ [7615233] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:13 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615233/ [7615234] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:27:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615234/ [7615235] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615235/ [7615236] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:39:39 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615236/ [7615237] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:12:33 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615237/ [7615238] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_adoption} time: 00:13:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615238/ [7615239] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615239/ [7615240] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615240/ [7615241] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep} time: 00:16:34 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615241/ [7615242] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:47 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615242/ [7615243] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:35:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615243/ [7615244] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:08 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615244/ [7615246] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:19:53 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615246/ [7615247] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615247/ [7615248] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/readwrite} time: 00:11:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615248/ [7615250] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:18:15 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615250/ [7615251] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:19:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615251/ [7615252] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:12:18 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615252/ [7615253] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:15:16 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615253/ [7615254] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} time: 00:14:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615254/ [7615255] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:19:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615255/ [7615256] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:21:26 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615256/ [7615257] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/e2e} time: 00:23:33 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615257/ [7615258] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:13 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615258/ [7615259] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615259/ [7615260] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615260/ [7615261] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_osdmap_prune} time: 00:52:42 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615261/ [7615262] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:31:13 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615262/ [7615263] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:10:35 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615263/ [7615264] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} time: 00:19:58 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615264/ [7615265] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615265/ [7615266] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:30:16 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615266/ [7615267] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:24 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615267/ [7615268] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:28 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615268/ [7615269] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:15:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615269/ [7615270] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:07:27 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615270/ [7615272] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/repair_test} time: 00:14:54 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615272/ [7615274] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} time: 00:15:16 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615274/ [7615275] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:26 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615275/ [7615276] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615276/ [7615277] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/lockdep} time: 00:12:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615277/ [7615278] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:12:55 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615278/ [7615279] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:17:59 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615279/ [7615280] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:54 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615280/ [7615281] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:45 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615281/ [7615282] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/insights} time: 00:11:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615282/ [7615285] rados/singleton/{all/deduptool mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:12:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615285/ [7615286] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:14:48 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615286/ [7615287] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:25:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615287/ [7615288] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615288/ [7615289] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} time: 00:13:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615289/ [7615290] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:08:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615290/ [7615291] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615291/ [7615292] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:25:19 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615292/ [7615293] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_mon_workunits} time: 00:27:39 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615293/ [7615294] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615294/ [7615295] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615295/ [7615297] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:25:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615297/ [7615298] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:36 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615298/ [7615299] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:27:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615299/ [7615300] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:27:38 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615300/ [7615301] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} time: 00:18:12 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615301/ [7615302] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:12:12 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615302/ [7615303] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:38 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615303/ [7615304] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:10:31 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615304/ [7615305] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:25:27 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615305/ [7615306] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615306/ [7615307] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:07 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615307/ [7615308] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:05 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615308/ [7615309] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:24:20 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615309/ [7615312] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:51 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615312/ [7615313] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:23:29 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615313/ [7615315] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:38:12 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615315/ [7615316] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:25 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615316/ [7615317] rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:16:49 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615317/ [7615318] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:26:20 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615318/ [7615319] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:18:34 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615319/ [7615320] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:13:24 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615320/ [7615321] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:20:00 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615321/ [7615322] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:31:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615322/ [7615323] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:48 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615323/ [7615324] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:28 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615324/ [7615325] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615325/ [7615326] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:24:37 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615326/ [7615327] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/snaps-few-objects} time: 00:29:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615327/ [7615329] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615329/ [7615330] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:15:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615330/ [7615331] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:34:11 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615331/ [7615332] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli} time: 00:13:41 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615332/ [7615333] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:09:56 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615333/ [7615334] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:45:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615334/ [7615335] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:08:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615335/ [7615337] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} time: 00:25:39 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615337/ [7615339] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:46 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615339/ [7615340] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:11:29 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615340/ [7615341] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:03 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615341/ [7615342] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:48 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615342/ [7615343] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:13:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615343/ [7615344] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:35:23 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615344/ [7615345] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615345/ [7615346] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:16:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615346/ [7615347] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:29:11 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615347/ [7615348] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:24:35 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615348/ [7615349] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:37:28 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615349/ [7615350] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:28:19 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615350/ [7615351] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:23 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615351/ [7615352] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:05 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615352/ [7615353] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615353/ [7615354] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:13:47 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615354/ [7615356] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:25:41 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615356/ [7615357] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:07:47 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615357/ [7615358] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:18 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615358/ [7615359] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:09:55 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615359/ [7615362] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:31:44 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615362/ [7615363] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:13:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615363/ [7615364] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:37 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615364/ [7615365] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:08:39 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615365/ [7615366] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615366/ [7615367] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} time: 00:15:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615367/ [7615368] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615368/ [7615369] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:15:40 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615369/ [7615370] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/progress} time: 00:19:27 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615370/ [7615371] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:50 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615371/ [7615372] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:24:36 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615372/ [7615374] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615374/ [7615375] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615375/ [7615376] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:13:30 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615376/ [7615377] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:56 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615377/ [7615378] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:26:59 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615378/ [7615380] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} time: 00:12:17 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615380/ [7615381] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:13:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615381/ [7615382] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:25 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615382/ [7615383] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:18 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615383/ [7615384] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:17:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615384/ [7615385] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:28:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615385/ [7615386] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:40 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615386/ [7615387] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:12:37 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615387/ [7615388] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/rados_python} time: 00:14:12 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615388/ [7615389] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:13:30 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615389/ [7615390] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615390/ [7615391] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:34 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615391/ [7615392] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:14 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615392/ [7615393] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:15:37 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615393/ [7615394] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:22:17 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615394/ [7615396] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/many workloads/pool-create-delete} time: 00:22:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615396/ [7615397] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} time: 02:25:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615397/ [7615398] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 00:28:54 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615398/ [7615399] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:11:49 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615399/ [7615400] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:16:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615400/ [7615401] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:39 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615401/ [7615402] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:50 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615402/ [7615403] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} time: 00:15:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615403/ [7615405] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:15:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615405/ [7615406] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615406/ [7615407] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615407/ [7615408] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:03 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615408/ [7615409] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:19:26 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615409/ [7615410] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:14:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615410/ [7615411] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:29:55 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615411/ [7615412] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/e2e} time: 00:25:15 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615412/ [7615413] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:23:36 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615413/ [7615415] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:21:16 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615415/ [7615416] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:13:58 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615416/ [7615417] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:14:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615417/ [7615418] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:11 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615418/ [7615419] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} time: 00:19:55 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615419/ [7615420] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:13:17 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615420/ [7615421] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615421/ [7615422] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:19:16 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615422/ [7615424] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 02:41:14 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615424/ [7615425] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:27 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615425/ [7615426] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:24:42 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615426/ [7615427] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:12:42 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615427/ [7615428] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:23 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615428/ [7615429] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:51 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615429/ [7615430] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_5925} time: 00:09:18 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615430/ [7615431] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:07 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615431/ [7615432] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:14:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615432/ [7615433] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:12:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615433/ [7615434] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:59 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615434/ [7615436] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:18 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615436/ [7615437] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:08:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615437/ [7615438] rados/singleton/{all/peer mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:08:18 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615438/ [7615440] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:11 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615440/ [7615441] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:50 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615441/ [7615442] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:38 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615442/ [7615443] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:16:03 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615443/ [7615444] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} time: 00:12:10 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615444/ [7615445] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} time: 00:22:53 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615445/ [7615447] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:18:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615447/ [7615448] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:07 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615448/ [7615449] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:20:59 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615449/ [7615450] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:26 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615450/ [7615451] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:18:37 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615451/ [7615452] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:17:57 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615452/ [7615453] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615453/ [7615454] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615454/ [7615456] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:11:56 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615456/ [7615458] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:26 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615458/ [7615459] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:19 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615459/ [7615460] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:03 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615460/ [7615461] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:45 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615461/ [7615462] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:19:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615462/ [7615463] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests} time: 00:27:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615463/ [7615464] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:10:21 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615464/ [7615465] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:44 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615465/ [7615466] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:19:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615466/ [7615467] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:18:53 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615467/ [7615468] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:17:13 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615468/ [7615469] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:26:35 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615469/ [7615471] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:29:25 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615471/ [7615472] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:44 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615472/ [7615473] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615473/ [7615474] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:22:24 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615474/ [7615476] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:44 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615476/ [7615477] rados/singleton/{all/random-eio mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:25:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615477/ [7615478] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:10:47 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615478/ [7615479] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615479/ [7615480] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:34:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615480/ [7615481] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli_mon} time: 00:24:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615481/ [7615482] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} time: 00:19:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615482/ [7615483] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/failover} time: 00:13:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615483/ [7615484] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:31:25 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615484/ [7615485] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:15:08 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615485/ [7615486] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/rbd_cls} time: 00:25:01 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615486/ [7615487] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:10:41 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615487/ [7615488] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_latest}} time: 00:36:09 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615488/ [7615489] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:12:47 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615489/ [7615491] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:35 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615491/ [7615493] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:22:19 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615493/ [7615495] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:13:32 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615495/ [7615496] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:16:27 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615496/ [7615497] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:05 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615497/ [7615498] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:09:42 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615498/ [7615499] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:36 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615499/ [7615500] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:23:04 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615500/ [7615501] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:44:14 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615501/ [7615502] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:24:38 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615502/ [7615503] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:14 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615503/ [7615504] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:15:52 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615504/ [7615505] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:11:02 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615505/ [7615506] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:34 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615506/ [7615507] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:34 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615507/ [7615508] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:26:51 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615508/ [7615509] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:43 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615509/ [7615510] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:22 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615510/ [7615511] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:27:07 info: https://pulpito.ceph.com/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615511/