2024-04-15T10:56:43.196 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-15T10:56:43.196 INFO:teuthology.results:subset: '17088/100000' 2024-04-15T10:56:43.196 INFO:teuthology.results:seed: '9898' 2024-04-15T14:10:08.002 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-15T14:10:08.029 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-16T00:56:09.699 INFO:teuthology.results:Tests finished! gathering results... 2024-04-16T00:56:34.559 INFO:scrape:Found 308 jobs 2024-04-16T00:56:42.508 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.508 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.508 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.527 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.527 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.527 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.543 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.543 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.543 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.560 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.560 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:56:42.560 WARNING:scrape:Saw end of BT but not start 2024-04-16T00:57:04.435 INFO:scrape:Found 14 distinct failure reasons 2024-04-16T00:57:04.436 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-16T00:57:04.436 INFO:scrape:5 jobs: ['7644400', '7644372', '7644262', '7644126', '7644334'] 2024-04-16T00:57:04.436 INFO:scrape:suites intersection: [] 2024-04-16T00:57:04.436 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-04-16T00:57:04.436 INFO:scrape: 2024-04-16T00:57:04.436 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2024-04-16T00:57:04.436 INFO:scrape:4 jobs: ['7644244', '7644322', '7644395', '7644171'] 2024-04-16T00:57:04.437 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool'] 2024-04-16T00:57:04.437 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-16T00:57:04.437 INFO:scrape: 2024-04-16T00:57:04.437 INFO:scrape:Failure: Failed to reconnect to smithi155 2024-04-16T00:57:04.437 INFO:scrape:1 jobs: ['7644173'] 2024-04-16T00:57:04.437 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/repave-all}', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc'] 2024-04-16T00:57:04.437 INFO:scrape: 2024-04-16T00:57:04.437 INFO:scrape:Failure: "2024-04-15T12:29:20.972036+0000 mon.a (mon.0) 613 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log 2024-04-16T00:57:04.437 INFO:scrape:45 jobs 2024-04-16T00:57:04.438 INFO:scrape:suites intersection: [] 2024-04-16T00:57:04.438 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/pacific', '1-install/reef', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/rgw', '3-final}', 'agent/off', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'debug/mgr', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/centos_9.stream_runc', 'rados/cephadm/smoke/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/dashboard/{0-single-container-host', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'random-objectstore$/{bluestore-comp-snappy}', 'random-objectstore$/{bluestore-comp-zstd}', 'rbd_import_export', 'start}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/e2e}', 'test_iscsi_container}}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/careful', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2024-04-16T00:57:04.438 INFO:scrape: 2024-04-16T00:57:04.438 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-04-16T00:57:04.438 INFO:scrape:1 jobs: ['7644309'] 2024-04-16T00:57:04.438 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-04-16T00:57:04.439 INFO:scrape: 2024-04-16T00:57:04.439 INFO:scrape:Failure: Command failed on smithi059 with status 5: 'sudo systemctl stop ceph-0281b196-fb34-11ee-bc8f-c7b262605968@mon.a' 2024-04-16T00:57:04.439 INFO:scrape:2 jobs: ['7644409', '7644172'] 2024-04-16T00:57:04.439 INFO:scrape:suites intersection: ['mon_election/connectivity'] 2024-04-16T00:57:04.439 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'agent/on', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'task/test_ca_signed_key}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps}'] 2024-04-16T00:57:04.439 INFO:scrape: 2024-04-16T00:57:04.439 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2024-04-16T00:57:04.439 INFO:scrape:1 jobs: ['7644196'] 2024-04-16T00:57:04.439 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-04-16T00:57:04.440 INFO:scrape: 2024-04-16T00:57:04.440 INFO:scrape:Dead: b'2024-04-15T23:08:41.682 DEBUG:teuthology.exit:Finished running handlers' 2024-04-16T00:57:04.440 INFO:scrape:2 jobs: ['7644114', '7644269'] 2024-04-16T00:57:04.440 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/admin_socket_output'] 2024-04-16T00:57:04.440 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-16T00:57:04.440 INFO:scrape: 2024-04-16T00:57:04.440 INFO:scrape:Failure: 2024-04-16T00:57:04.440 INFO:scrape:4 jobs: ['7644303', '7644370', '7644164', '7644232'] 2024-04-16T00:57:04.440 INFO:scrape:suites intersection: ['ceph', 'mon_election/connectivity', 'rados', 'rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}'] 2024-04-16T00:57:04.440 INFO:scrape:suites union: ['ceph', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}'] 2024-04-16T00:57:04.441 INFO:scrape: 2024-04-16T00:57:04.441 INFO:scrape:Failure: Command failed on smithi073 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" 2024-04-16T00:57:04.441 INFO:scrape:3 jobs: ['7644259', '7644203', '7644360'] 2024-04-16T00:57:04.441 INFO:scrape:suites intersection: ['rados'] 2024-04-16T00:57:04.441 INFO:scrape:suites union: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/octopus', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/rbd_cls}'] 2024-04-16T00:57:04.441 INFO:scrape: 2024-04-16T00:57:04.441 INFO:scrape:Crash: Test failure: test_turn_off_module (tasks.mgr.test_progress.TestProgress) 2024-04-16T00:57:04.441 INFO:scrape:ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5b57da8520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x556c0c1010a2] 6: (KernelDevice::_aio_thread()+0x9e2) [0x556c0cabc462] 7: ceph-osd(+0xd90251) [0x556c0ca98251] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5b57dfab43] 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5b57e8ca00] 2024-04-16T00:57:04.441 INFO:scrape:1 jobs: ['7644153'] 2024-04-16T00:57:04.441 INFO:scrape:suites: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/connectivity', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-stupid}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/progress}'] 2024-04-16T00:57:04.442 INFO:scrape: 2024-04-16T00:57:04.442 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-16T00:57:04.442 INFO:scrape:3 jobs: ['7644296', '7644162', '7644414'] 2024-04-16T00:57:04.442 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2024-04-16T00:57:04.442 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2024-04-16T00:57:04.442 INFO:scrape: 2024-04-16T00:57:04.442 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-04-16T00:57:04.442 INFO:scrape:1 jobs: ['7644237'] 2024-04-16T00:57:04.442 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}'] 2024-04-16T00:57:04.442 INFO:scrape: 2024-04-16T00:57:04.443 INFO:scrape:Failure: Command failed (workunit test mon/mkfs.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mkfs.sh' 2024-04-16T00:57:04.443 INFO:scrape:1 jobs: ['7644145'] 2024-04-16T00:57:04.443 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/mon}'] 2024-04-16T00:57:04.443 INFO:scrape: 2024-04-16T00:57:04.456 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/ failed: 72 dead: 2 running: 0 waiting: 0 queued: 0 passed: 234 Fail ================================================================= [7644110] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:14:11 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644110/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644110/ "2024-04-15T11:17:30.942830+0000 mon.smithi055 (mon.0) 787 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644116] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 02:04:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644116/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644116/ "2024-04-15T11:42:59.453646+0000 mon.a (mon.0) 331 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644119] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:13:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644119/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644119/ "2024-04-15T11:26:43.043972+0000 mon.a (mon.0) 211 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644126] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:31:19 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644126/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644126/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7644132] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644132/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644132/ "2024-04-15T11:42:23.426843+0000 mon.smithi162 (mon.0) 759 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644136] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:15:04 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644136/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644136/ "2024-04-15T11:38:50.195428+0000 mon.a (mon.0) 416 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7644144] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:24:17 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644144/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644144/ "2024-04-15T11:44:37.815190+0000 mon.a (mon.0) 213 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644145] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} ----------------------------------------------------------------- time: 00:12:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644145/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644145/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15d92d1715a946b1be5b15f80f65ad8c Command failed (workunit test mon/mkfs.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mkfs.sh' [7644153] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:23:07 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644153/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b508fad8d5c64a348b2e47d5cf2b615c Test failure: test_turn_off_module (tasks.mgr.test_progress.TestProgress) [7644157] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:30 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644157/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644157/ "2024-04-15T11:57:19.866749+0000 mon.smithi144 (mon.0) 772 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644162] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:17:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644162/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644162/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c23f61791bbc4544aac5b4529f1f16d5 Command failed (workunit test cephadm/test_cephadm.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7644164] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644164/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644164/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88bc8a3d575147eb8beb894f1473ae62 [7644168] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:31 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644168/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644168/ "2024-04-15T12:02:41.263963+0000 mon.a (mon.0) 665 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7644171] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:26:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644171/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644171/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6234fc0c3e734907a724634230bd96f0 Command failed (workunit test cephtool/test.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7644172] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644172/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9bf602879e84ba687ecaa282f37a7e7 Command failed on smithi155 with status 1: 'sudo yum install -y kernel' [7644173] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644173/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644173/ sentry: https://sentry.ceph.com/organizations/ceph/?query=882bd07c0c77403cad10b5e683739b8a Failed to reconnect to smithi155 [7644183] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:34:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644183/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644183/ "2024-04-15T12:20:00.000096+0000 mon.a (mon.0) 1403 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [7644191] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:13:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644191/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644191/ "2024-04-15T12:15:00.397341+0000 mon.a (mon.0) 297 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7644194] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/e2e} ----------------------------------------------------------------- time: 00:25:20 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644194/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644194/ "2024-04-15T12:29:20.972036+0000 mon.a (mon.0) 613 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7644196] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 03:19:36 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644196/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77920e2a40054c9199bf4a900b1f4a87 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi123 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7644197] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:33 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644197/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644197/ "2024-04-15T12:24:22.449587+0000 mon.smithi032 (mon.0) 777 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644203] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644203/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79f51041fc2144e48f6aa4336fff985c Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7644205] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:23:16 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644205/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644205/ "2024-04-15T12:31:10.189999+0000 mon.a (mon.0) 649 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644213] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:25:35 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644213/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644213/ "2024-04-15T12:25:41.663785+0000 mon.a (mon.0) 251 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log [7644218] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:37 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644218/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644218/ "2024-04-15T12:33:35.803313+0000 mon.smithi059 (mon.0) 786 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644222] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:23:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644222/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644222/ "2024-04-15T12:50:00.000095+0000 mon.a (mon.0) 1417 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [7644226] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:13:43 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644226/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644226/ "2024-04-15T12:32:15.595431+0000 mon.a (mon.0) 191 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7644232] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644232/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af83511e3e1d4dceae157dd2e302bc6b [7644233] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:19:06 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644233/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644233/ "2024-04-15T12:41:42.407144+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644237] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:51:14 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644237/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644237/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb6c0ab5416a4db1a251a87227680e6b Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7644238] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:14:18 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644238/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644238/ "2024-04-15T12:42:31.202814+0000 mon.smithi165 (mon.0) 720 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7644243] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:20:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644243/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644243/ "2024-04-15T12:51:35.845457+0000 mon.a (mon.0) 434 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7644244] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:28:06 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644244/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644244/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29a81189b3bc45feb1cf3c05cfe75f87 Command failed (workunit test cephtool/test.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7644250] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:22 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644250/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644250/ "2024-04-15T12:45:49.664852+0000 mon.a (mon.0) 370 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7644257] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:21:19 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644257/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644257/ "2024-04-15T13:05:08.109315+0000 mon.smithi066 (mon.0) 762 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644259] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:30:49 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644259/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644259/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2213729dea304a9487798617942c84c0 Command failed on smithi073 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-Te stClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" [7644262] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:50:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644262/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644262/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7644263] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:16:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644263/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644263/ "2024-04-15T13:05:10.307674+0000 mon.a (mon.0) 207 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644271] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:46:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644271/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644271/ "2024-04-15T13:13:48.142919+0000 mon.a (mon.0) 369 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644293] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:14:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644293/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644293/ "2024-04-15T13:22:39.306477+0000 mon.a (mon.0) 540 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7644296] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:13:18 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644296/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d9ff7c815b94d94be5b819acd61be30 Command failed (workunit test cephadm/test_cephadm.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7644301] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:40:28 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644301/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644301/ "2024-04-15T13:32:12.730541+0000 mon.a (mon.0) 214 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644303] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644303/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644303/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f2c31564ba24aec88d617fb137e861a [7644305] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644305/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644305/ "2024-04-15T13:28:44.893222+0000 mon.smithi031 (mon.0) 773 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644309] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644309/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644309/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20ee1f1576a34e0d8a84c8d758e9f350 Command failed (workunit test cephadm/test_repos.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7644313] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:19:48 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644313/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644313/ "2024-04-15T13:38:12.584934+0000 mon.smithi006 (mon.0) 765 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644322] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:25:52 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644322/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644322/ sentry: https://sentry.ceph.com/organizations/ceph/?query=672f1373ec47432db40b6fec6418faa8 Command failed (workunit test cephtool/test.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7644324] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:13:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644324/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644324/ "2024-04-15T13:35:19.718685+0000 mon.a (mon.0) 350 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7644329] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:25:59 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644329/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644329/ "2024-04-15T13:43:39.213950+0000 mon.a (mon.0) 381 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7644334] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:29:35 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644334/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644334/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7644336] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:16:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644336/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644336/ "2024-04-15T13:46:59.171256+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644341] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644341/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644341/ "2024-04-15T13:51:21.625825+0000 mon.smithi145 (mon.0) 795 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644345] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/e2e} ----------------------------------------------------------------- time: 00:27:16 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644345/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644345/ "2024-04-15T14:04:55.102624+0000 mon.a (mon.0) 1174 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7644347] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:14:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644347/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644347/ "2024-04-14T08:40:01.557720+0000 mon.a (mon.0) 322 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7644352] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:12:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644352/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644352/ "2024-04-15T13:50:49.982499+0000 mon.smithi161 (mon.0) 449 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7644360] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:03 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644360/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee5f98269a8f487f913b780df7da7cd6 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7644362] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:15:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644362/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644362/ "2024-04-15T14:03:55.977307+0000 mon.smithi062 (mon.0) 789 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644368] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:19:13 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644368/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644368/ "2024-04-15T14:04:03.514975+0000 mon.a (mon.0) 210 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644370] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:00:02 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644370/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=641834eb8827489494d1c45fc3340761 [7644372] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:18:42 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644372/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644372/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7644373] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:14:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644373/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644373/ "2024-04-15T14:01:06.510275+0000 mon.a (mon.0) 192 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log [7644376] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:34:25 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644376/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644376/ "2024-04-15T14:08:36.175032+0000 mon.a (mon.0) 206 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644380] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:14:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644380/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644380/ "2024-04-15T14:09:27.495953+0000 mon.a (mon.0) 266 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644384] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:18:52 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644384/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644384/ "2024-04-15T14:16:40.171293+0000 mon.smithi115 (mon.0) 555 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7644392] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:21:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644392/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644392/ "2024-04-15T14:23:30.680436+0000 mon.a (mon.0) 215 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644395] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:28:59 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644395/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=106f2c33ed8041d78abbcaf539b24b50 Command failed (workunit test cephtool/test.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7644397] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:14:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644397/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644397/ "2024-04-15T14:23:24.173006+0000 mon.smithi042 (mon.0) 795 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7644400] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:46:15 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644400/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644400/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7644404] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:26 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644404/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644404/ "2024-04-15T14:20:29.913072+0000 mon.a (mon.0) 483 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7644409] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:14 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644409/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff64ebc607a84e31a3e9c30006d628db Command failed on smithi059 with status 5: 'sudo systemctl stop ceph-0281b196-fb34-11ee-bc8f-c7b262605968@mon.a' [7644412] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:32:05 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644412/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644412/ "2024-04-15T14:28:57.011603+0000 mon.a (mon.0) 215 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7644414] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:13:13 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644414/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d77ea589ea5243c9a5177c32f5e5880a Command failed (workunit test cephadm/test_cephadm.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a971fd5f5719038c70b817bafacd6d5400b1d888 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' Dead ================================================================= [7644114] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644114/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644114/ hit max job timeout [7644269] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644269/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644269/ hit max job timeout Pass ================================================================= [7644108] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:45:20 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644108/ [7644109] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:48:54 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644109/ [7644111] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:13:25 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644111/ [7644112] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644112/ [7644113] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} time: 01:11:32 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644113/ [7644115] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} time: 00:23:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644115/ [7644117] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:22:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644117/ [7644118] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:35:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644118/ [7644120] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:22:11 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644120/ [7644121] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:29 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644121/ [7644122] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:44 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644122/ [7644123] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:09 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644123/ [7644124] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:31 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644124/ [7644125] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:12:33 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644125/ [7644127] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:21:38 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644127/ [7644128] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:14:05 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644128/ [7644129] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:13:12 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644129/ [7644130] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} time: 00:20:23 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644130/ [7644131] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:11:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644131/ [7644133] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:24:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644133/ [7644134] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:10:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644134/ [7644135] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} time: 02:08:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644135/ [7644137] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:59 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644137/ [7644138] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:32:11 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644138/ [7644139] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:17:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644139/ [7644140] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:26:58 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644140/ [7644141] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:08:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644141/ [7644142] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_set_mon_crush_locations} time: 00:16:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644142/ [7644143] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_mon_workunits} time: 00:27:18 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644143/ [7644146] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644146/ [7644147] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:12 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644147/ [7644148] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:29:11 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644148/ [7644149] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:16 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644149/ [7644150] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_ca_signed_key} time: 00:11:22 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644150/ [7644151] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:24:31 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644151/ [7644152] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:21:07 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644152/ [7644154] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644154/ [7644155] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:45 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644155/ [7644156] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:15 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644156/ [7644158] rados/singleton/{all/deduptool mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:12:47 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644158/ [7644159] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/lockdep} time: 00:25:39 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644159/ [7644160] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:22:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644160/ [7644161] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} time: 00:46:36 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644161/ [7644163] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644163/ [7644165] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:14:48 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644165/ [7644166] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:18:47 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644166/ [7644167] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644167/ [7644169] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:25 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644169/ [7644170] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:38 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644170/ [7644174] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:09:31 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644174/ [7644175] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:48:19 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644175/ [7644176] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:28:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644176/ [7644177] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:38 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644177/ [7644178] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_latest}} time: 00:06:34 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644178/ [7644179] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_cephadm_repos} time: 00:06:22 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644179/ [7644180] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:07:42 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644180/ [7644181] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:33 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644181/ [7644182] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:13:04 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644182/ [7644184] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:12 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644184/ [7644185] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} time: 00:18:58 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644185/ [7644186] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:21:59 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644186/ [7644187] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:11 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644187/ [7644188] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:06 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644188/ [7644189] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:11:52 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644189/ [7644190] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:18 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644190/ [7644192] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:58 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644192/ [7644193] rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:18:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644193/ [7644195] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-radosbench} time: 00:59:35 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644195/ [7644198] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} time: 00:12:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644198/ [7644199] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:22 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644199/ [7644200] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} time: 00:19:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644200/ [7644201] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:31:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644201/ [7644202] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:36 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644202/ [7644204] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} time: 00:15:29 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644204/ [7644206] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 02:44:43 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644206/ [7644207] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:10:00 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644207/ [7644208] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:23 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644208/ [7644209] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:09:53 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644209/ [7644210] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:55 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644210/ [7644211] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:03 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644211/ [7644212] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete} time: 00:19:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644212/ [7644214] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:32:53 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644214/ [7644215] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:12:32 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644215/ [7644216] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:55 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644216/ [7644217] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644217/ [7644219] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:13:48 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644219/ [7644220] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:05 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644220/ [7644221] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:04 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644221/ [7644223] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644223/ [7644224] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:27:37 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644224/ [7644225] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:10:24 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644225/ [7644227] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-overwrites} time: 00:46:06 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644227/ [7644228] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:04 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644228/ [7644229] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/lockdep} time: 00:11:32 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644229/ [7644230] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:30:15 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644230/ [7644231] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:39 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644231/ [7644234] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:34 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644234/ [7644235] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:23:24 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644235/ [7644236] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:03 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644236/ [7644239] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:32:06 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644239/ [7644240] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:12:14 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644240/ [7644241] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:44 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644241/ [7644242] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:43 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644242/ [7644245] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} time: 00:11:17 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644245/ [7644246] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:13:55 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644246/ [7644247] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:12 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644247/ [7644248] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:35 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644248/ [7644249] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 00:31:14 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644249/ [7644251] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644251/ [7644252] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} time: 00:23:02 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644252/ [7644253] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:17 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644253/ [7644254] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:18 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644254/ [7644255] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:35 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644255/ [7644256] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:11:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644256/ [7644258] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:33:39 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644258/ [7644260] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:29 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644260/ [7644261] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:12:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644261/ [7644264] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:10:55 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644264/ [7644265] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} time: 00:19:37 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644265/ [7644266] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} time: 01:11:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644266/ [7644267] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_latest}} time: 00:35:02 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644267/ [7644268] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:13:09 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644268/ [7644270] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:45 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644270/ [7644272] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:18:42 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644272/ [7644273] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:15:47 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644273/ [7644274] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644274/ [7644275] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:28:06 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644275/ [7644276] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:16:16 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644276/ [7644277] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:11:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644277/ [7644278] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:24 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644278/ [7644279] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644279/ [7644280] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:24 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644280/ [7644281] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:16:13 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644281/ [7644282] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests} time: 00:27:13 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644282/ [7644283] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:15:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644283/ [7644284] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:42 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644284/ [7644285] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:13:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644285/ [7644286] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:21:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644286/ [7644287] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:30 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644287/ [7644288] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644288/ [7644289] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:12:09 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644289/ [7644290] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:14:26 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644290/ [7644291] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:24 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644291/ [7644292] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644292/ [7644294] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:04 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644294/ [7644295] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:15:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644295/ [7644297] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:16:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644297/ [7644298] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/lockdep} time: 00:17:32 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644298/ [7644299] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:22 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644299/ [7644300] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/insights} time: 00:11:28 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644300/ [7644302] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:20:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644302/ [7644304] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644304/ [7644306] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:12:24 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644306/ [7644307] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/repair_test} time: 00:15:14 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644307/ [7644308] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:23:57 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644308/ [7644310] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644310/ [7644311] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_latest}} time: 00:09:42 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644311/ [7644312] rados/singleton/{all/peer mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:09:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644312/ [7644314] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:27:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644314/ [7644315] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:44:18 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644315/ [7644316] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:22:05 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644316/ [7644317] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:31 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644317/ [7644318] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:20:34 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644318/ [7644319] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:22:41 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644319/ [7644320] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644320/ [7644321] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:13:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644321/ [7644323] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:16:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644323/ [7644325] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644325/ [7644326] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644326/ [7644327] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:43 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644327/ [7644328] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:26:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644328/ [7644330] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/scrub_test} time: 00:10:33 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644330/ [7644331] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:23:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644331/ [7644332] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:17:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644332/ [7644333] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} time: 01:21:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644333/ [7644335] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:20 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644335/ [7644337] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} time: 00:21:54 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644337/ [7644338] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:10:43 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644338/ [7644339] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:20:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644339/ [7644340] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:24:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644340/ [7644342] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:31 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644342/ [7644343] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:19 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644343/ [7644344] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:13:52 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644344/ [7644346] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:42:32 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644346/ [7644348] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:29:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644348/ [7644349] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:27 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644349/ [7644350] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:59 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644350/ [7644351] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:16:14 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644351/ [7644353] rados/multimon/{clusters/3 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644353/ [7644354] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:24:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644354/ [7644355] rados/singleton/{all/random-eio mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:24:54 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644355/ [7644356] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats} time: 00:09:46 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644356/ [7644357] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:12 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644357/ [7644358] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:43 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644358/ [7644359] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} time: 00:12:38 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644359/ [7644361] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:21:25 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644361/ [7644363] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} time: 00:15:13 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644363/ [7644364] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:12:48 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644364/ [7644365] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:09 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644365/ [7644366] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:27:50 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644366/ [7644367] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/lockdep} time: 00:25:34 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644367/ [7644369] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:34 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644369/ [7644371] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644371/ [7644374] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:09 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644374/ [7644375] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:15:42 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644375/ [7644377] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:29:52 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644377/ [7644378] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:25:17 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644378/ [7644379] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:09:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644379/ [7644381] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:08 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644381/ [7644382] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete} time: 00:21:01 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644382/ [7644383] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:09 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644383/ [7644385] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:19:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644385/ [7644386] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/progress} time: 00:20:52 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644386/ [7644387] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:10:48 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644387/ [7644388] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:07:48 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644388/ [7644389] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:47 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644389/ [7644390] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:28 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644390/ [7644391] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:18:44 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644391/ [7644393] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:13:44 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644393/ [7644394] rados/singleton/{all/test-noautoscale-flag mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:08:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644394/ [7644396] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:18:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644396/ [7644398] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:58 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644398/ [7644399] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} time: 00:16:13 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644399/ [7644401] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:51 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644401/ [7644402] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:21:12 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644402/ [7644403] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} time: 00:21:59 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644403/ [7644405] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644405/ [7644406] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:36 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644406/ [7644407] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:14:40 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644407/ [7644408] rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:24:07 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644408/ [7644410] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:49:23 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644410/ [7644411] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:10:56 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644411/ [7644413] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:49 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644413/ [7644415] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} time: 00:11:10 info: https://pulpito.ceph.com/teuthology-2024-04-07_20:00:14-rados-main-distro-default-smithi/7644415/