2024-03-14T23:42:48.036 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-14T23:42:48.037 INFO:teuthology.results:subset: '111/120000' 2024-03-14T23:42:48.037 INFO:teuthology.results:seed: '8735' 2024-03-15T03:56:30.867 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-15T03:56:30.895 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-15T15:33:35.447 INFO:teuthology.results:Tests finished! gathering results... 2024-03-15T15:33:58.491 INFO:scrape:Found 294 jobs 2024-03-15T15:33:59.321 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600671/teuthology.log 2024-03-15T15:34:11.553 INFO:scrape:Found 12 distinct failure reasons 2024-03-15T15:34:11.553 INFO:scrape:Failure: "2024-03-15T03:31:32.144124+0000 osd.0 (osd.0) 6 : cluster [ERR] osd.0 pg[3.13]: reservation requested while still reserved" in cluster log 2024-03-15T15:34:11.553 INFO:scrape:39 jobs 2024-03-15T15:34:11.554 INFO:scrape:suites intersection: [] 2024-03-15T15:34:11.554 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/pacific', '1-install/quincy', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '1-start', '1-tasks', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '2-services/basic', '2-services/rgw', '3-final}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'agent/off', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'clusters/{openstack', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-balancer/read', 'd-balancer/upmap-read', 'debug/mgr', 'fast/fast', 'fast/normal', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04}', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/dashboard/{0-single-container-host', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'random-objectstore$/{bluestore-comp-snappy}', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'rbd_import_export', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'start}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'task/test_host_drain}', 'task/test_monitoring_stack_basic}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/e2e}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/cache-agent-big}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}', 'workloads/write_fadvise_dontneed}'] 2024-03-15T15:34:11.554 INFO:scrape: 2024-03-15T15:34:11.554 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-03-15T15:34:11.554 INFO:scrape:2 jobs: ['7600827', '7600681'] 2024-03-15T15:34:11.554 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}'] 2024-03-15T15:34:11.554 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}'] 2024-03-15T15:34:11.555 INFO:scrape: 2024-03-15T15:34:11.555 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-15T15:34:11.555 INFO:scrape:5 jobs: ['7600693', '7600692', '7600560', '7600630', '7600758'] 2024-03-15T15:34:11.555 INFO:scrape:suites intersection: [] 2024-03-15T15:34:11.555 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-03-15T15:34:11.555 INFO:scrape: 2024-03-15T15:34:11.555 INFO:scrape:Dead: None 2024-03-15T15:34:11.555 INFO:scrape:3 jobs: ['7600671', '7600700', '7600599'] 2024-03-15T15:34:11.555 INFO:scrape: 2024-03-15T15:34:11.555 INFO:scrape:Failure: Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-316de02e-e261-11ee-95c9-87774f69a715@mon.a' 2024-03-15T15:34:11.555 INFO:scrape:1 jobs: ['7600566'] 2024-03-15T15:34:11.556 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-03-15T15:34:11.556 INFO:scrape: 2024-03-15T15:34:11.556 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-15T15:34:11.556 INFO:scrape:1 jobs: ['7600565'] 2024-03-15T15:34:11.556 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-03-15T15:34:11.556 INFO:scrape: 2024-03-15T15:34:11.556 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-03-15T15:34:11.556 INFO:scrape:1 jobs: ['7600577'] 2024-03-15T15:34:11.556 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] 2024-03-15T15:34:11.556 INFO:scrape: 2024-03-15T15:34:11.556 INFO:scrape:Dead: b'2024-03-15T13:22:55.471 DEBUG:teuthology.exit:Finished running handlers' 2024-03-15T15:34:11.557 INFO:scrape:6 jobs: ['7600663', '7600689', '7600745', '7600743', '7600801', '7600595'] 2024-03-15T15:34:11.557 INFO:scrape:suites intersection: [] 2024-03-15T15:34:11.557 INFO:scrape:suites union: ['0-nvme-loop', '0-start', '1-start', '1-tasks', '2-ops/rmdir-reactivate}', 'agent/on', 'clusters/3-mons', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async', 'objectstore/bluestore-bitmap', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/centos_9.stream_runc', 'rados/monthrash/{ceph', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'start}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/sync-many', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/pool-create-delete}'] 2024-03-15T15:34:11.557 INFO:scrape: 2024-03-15T15:34:11.557 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-03-15T15:34:11.557 INFO:scrape:1 jobs: ['7600570'] 2024-03-15T15:34:11.557 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-03-15T15:34:11.557 INFO:scrape: 2024-03-15T15:34:11.557 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2024-03-15T15:34:11.557 INFO:scrape:1 jobs: ['7600825'] 2024-03-15T15:34:11.558 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2024-03-15T15:34:11.558 INFO:scrape: 2024-03-15T15:34:11.558 INFO:scrape:Failure: "2024-03-15T00:50:08.687169+0000 mon.a (mon.0) 647 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2024-03-15T15:34:11.558 INFO:scrape:2 jobs: ['7600605', '7600796'] 2024-03-15T15:34:11.558 INFO:scrape:suites intersection: ['0-nvme-loop'] 2024-03-15T15:34:11.558 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', 'agent/off', 'fixed-2', 'mon_election/classic', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'start}'] 2024-03-15T15:34:11.558 INFO:scrape: 2024-03-15T15:34:11.558 INFO:scrape:Failure: "2024-03-15T00:50:00.000162+0000 mon.a (mon.0) 1767 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 296/2160 objects degraded (13.704%), 5 pgs degraded" in cluster log 2024-03-15T15:34:11.558 INFO:scrape:1 jobs: ['7600589'] 2024-03-15T15:34:11.558 INFO:scrape:suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/reef', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/default', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}'] 2024-03-15T15:34:11.558 INFO:scrape: 2024-03-15T15:34:11.568 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/ failed: 54 dead: 9 running: 0 waiting: 0 queued: 0 passed: 231 Fail ================================================================= [7600553] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:23:39 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600553/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600553/ "2024-03-15T00:04:54.715997+0000 mon.a (mon.0) 194 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600554] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:13:30 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600554/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600554/ "2024-03-15T00:05:01.121378+0000 mon.smithi107 (mon.0) 781 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600557] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:27:25 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600557/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600557/ "2024-03-15T00:15:37.602176+0000 osd.1 (osd.1) 17 : cluster [ERR] osd.1 pg[3.8s1]: reservation requested while still reserved" in cluster log [7600560] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:45:28 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600560/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600560/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7600565] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:35:49 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600565/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa3ed70118bc487b91c33225ddb4db1f Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7600566] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600566/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=24eef355aa614465801d22b5f9e5b57b Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-316de02e-e261-11ee-95c9-87774f69a715@mon.a' [7600569] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:42:29 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600569/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600569/ "2024-03-15T00:25:41.319064+0000 osd.3 (osd.3) 37 : cluster [ERR] osd.3 pg[1.es2]: reservation requested while still reserved" in cluster log [7600570] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 04:25:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600570/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600570/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a78cab6219a94621820623a6d47f8557 Command failed (workunit test rados/test.sh) on smithi045 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7600574] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:41:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600574/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600574/ "2024-03-15T00:23:54.933181+0000 osd.3 (osd.3) 146 : cluster [ERR] osd.3 pg[7.1]: reservation requested while still reserved" in cluster log [7600577] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:12:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600577/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe89ce9cb6f54605891089f00b23abfb Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7600589] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:35:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600589/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600589/ "2024-03-15T00:50:00.000162+0000 mon.a (mon.0) 1767 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 296/2160 objects degraded (13.704%), 5 pgs degraded" in cluster log [7600590] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:39 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600590/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600590/ "2024-03-15T00:36:05.256994+0000 mon.smithi012 (mon.0) 786 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600597] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:57:37 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600597/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600597/ "2024-03-15T00:43:16.068699+0000 mon.a (mon.0) 176 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7600605] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:55 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600605/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600605/ "2024-03-15T00:50:08.687169+0000 mon.a (mon.0) 647 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7600617] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600617/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600617/ "2024-03-15T00:58:03.924126+0000 mon.smithi149 (mon.0) 238 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600629] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:15:15 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600629/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600629/ "2024-03-15T01:11:31.198321+0000 mon.a (mon.0) 191 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600630] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:02:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600630/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600630/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7600639] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:43 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600639/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600639/ "2024-03-15T01:25:19.954443+0000 mon.smithi070 (mon.0) 766 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600653] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:22:18 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600653/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600653/ "2024-03-15T01:31:36.190671+0000 osd.0 (osd.0) 11 : cluster [ERR] osd.0 pg[3.fs2]: reservation requested while still reserved" in cluster log [7600654] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:13:45 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600654/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600654/ "2024-03-15T01:31:08.432000+0000 osd.4 (osd.4) 52 : cluster [ERR] osd.4 pg[3.f]: reservation requested while still reserved" in cluster log [7600664] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:25:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600664/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600664/ "2024-03-15T01:39:35.806492+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600669] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} tasks/e2e} ----------------------------------------------------------------- time: 00:28:09 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600669/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600669/ "2024-03-15T01:58:03.040980+0000 mon.a (mon.0) 1141 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600677] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:16:10 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600677/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600677/ "2024-03-15T01:50:51.060411+0000 mon.a (mon.0) 427 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7600681] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:10 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600681/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600681/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7630059e8b94a0fa9c7cb19e86b6c82 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7600685] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:21:10 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600685/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600685/ "2024-03-15T02:01:24.347344+0000 osd.5 (osd.5) 15 : cluster [WRN] slow reservation response from 6 (74599ms)" in cluster log [7600692] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:17:30 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600692/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600692/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7600693] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:32:21 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600693/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600693/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7600694] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:33:36 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600694/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600694/ "2024-03-15T02:16:15.675899+0000 osd.7 (osd.7) 98 : cluster [ERR] osd.7 pg[3.11s0]: reservation requested while still reserved" in cluster log [7600698] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:29:50 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600698/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600698/ "2024-03-15T02:12:55.790016+0000 osd.1 (osd.1) 11 : cluster [ERR] osd.1 pg[3.0]: reservation requested while still reserved" in cluster log [7600701] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:38:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600701/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600701/ "2024-03-15T02:10:17.177614+0000 mon.a (mon.0) 199 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600722] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:30:21 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600722/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600722/ "2024-03-15T02:40:56.537595+0000 osd.2 (osd.2) 287 : cluster [ERR] osd.2 pg[1.0]: reservation requested while still reserved" in cluster log [7600723] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:24:55 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600723/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600723/ "2024-03-15T02:31:36.110526+0000 mon.a (mon.0) 416 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7600729] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:32:13 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600729/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600729/ "2024-03-15T02:51:10.900901+0000 osd.3 (osd.3) 176 : cluster [ERR] osd.3 pg[319.5s2]: reservation requested while still reserved" in cluster log [7600736] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:22 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600736/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600736/ "2024-03-15T02:41:40.038016+0000 mon.smithi008 (mon.0) 754 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600749] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:17:07 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600749/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600749/ "2024-03-15T02:54:28.828840+0000 mon.a (mon.0) 189 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600753] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:26:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600753/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600753/ "2024-03-15T03:00:32.181127+0000 osd.5 (osd.5) 25 : cluster [ERR] osd.5 pg[3.as2]: reservation requested while still reserved" in cluster log [7600758] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:48:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600758/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600758/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7600761] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:24:54 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600761/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600761/ "2024-03-15T03:06:20.678797+0000 mon.a (mon.0) 182 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log [7600771] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:19:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600771/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600771/ "2024-03-15T03:15:11.032874+0000 mon.smithi129 (mon.0) 492 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7600775] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:39:11 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600775/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600775/ "2024-03-15T03:21:50.354807+0000 mon.a (mon.0) 186 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600776] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:26:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600776/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600776/ "2024-03-15T03:31:32.144124+0000 osd.0 (osd.0) 6 : cluster [ERR] osd.0 pg[3.13]: reservation requested while still reserved" in cluster log [7600784] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:24:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600784/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600784/ "2024-03-15T03:33:52.225534+0000 mon.a (mon.0) 359 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7600796] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:19:51 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600796/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600796/ "2024-03-15T03:40:49.216190+0000 mon.smithi008 (mon.0) 584 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7600806] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:29:01 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600806/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600806/ "2024-03-15T04:04:24.360550+0000 osd.4 (osd.4) 95 : cluster [ERR] osd.4 pg[3.1]: reservation requested while still reserved" in cluster log [7600811] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:24:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600811/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600811/ "2024-03-15T04:02:29.089736+0000 mon.a (mon.0) 196 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7600815] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:25:19 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600815/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600815/ "2024-03-15T04:01:05.399280+0000 osd.2 (osd.2) 272 : cluster [ERR] osd.2 pg[3.3s1]: reservation requested while still reserved" in cluster log [7600817] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} tasks/e2e} ----------------------------------------------------------------- time: 00:23:04 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600817/ "2024-03-15T04:12:51.345019+0000 mon.a (mon.0) 630 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600819] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:41:58 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600819/ "2024-03-15T04:12:33.817622+0000 osd.0 (osd.0) 25 : cluster [ERR] osd.0 pg[1.1s2]: reservation requested while still reserved" in cluster log [7600822] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:36 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600822/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600822/ "2024-03-15T04:07:31.953572+0000 mon.smithi032 (mon.0) 771 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7600825] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:02:17 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600825/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26e7525065914e9b85a9eafe99e6a6fb Command failed (workunit test rados/test.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8fb72a1057818c8e23ab8db51bece79404afc7d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7600826] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:41:17 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600826/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600826/ "2024-03-15T04:16:52.462296+0000 osd.5 (osd.5) 14 : cluster [ERR] osd.5 pg[4.2]: reservation requested while still reserved" in cluster log [7600827] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:12:53 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ebef5b8d3ed4fe4aced6fe74fc12bcc Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7600833] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:15:19 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600833/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600833/ "2024-03-15T04:11:52.401610+0000 mon.a (mon.0) 206 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log [7600837] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:22:38 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600837/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600837/ "2024-03-15T04:18:04.819491+0000 osd.4 (osd.4) 15 : cluster [ERR] osd.4 pg[4.1]: reservation requested while still reserved" in cluster log Dead ================================================================= [7600595] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600595/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600595/ hit max job timeout [7600599] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600599/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600599/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7600663] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600663/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600663/ hit max job timeout [7600671] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600671/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600671/ Error reimaging machines: Failed to power on smithi005 [7600689] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600689/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600689/ hit max job timeout [7600700] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600700/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600700/ Error reimaging machines: Failed to power on smithi149 [7600743] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600743/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600743/ hit max job timeout [7600745] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600745/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600745/ hit max job timeout [7600801] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600801/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600801/ hit max job timeout Pass ================================================================= [7600545] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:18:01 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600545/ [7600546] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:11:38 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600546/ [7600547] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:29 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600547/ [7600548] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600548/ [7600549] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} time: 00:15:43 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600549/ [7600550] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:49 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600550/ [7600551] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} time: 00:07:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600551/ [7600552] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:22:26 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600552/ [7600555] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600555/ [7600556] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600556/ [7600558] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:14:52 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600558/ [7600559] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:12:47 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600559/ [7600561] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:25 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600561/ [7600562] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:25:48 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600562/ [7600563] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:22:49 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600563/ [7600564] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:36 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600564/ [7600567] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:30 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600567/ [7600568] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:36:08 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600568/ [7600571] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:25 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600571/ [7600572] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:31:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600572/ [7600573] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:10:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600573/ [7600575] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:08:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600575/ [7600576] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:51 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600576/ [7600578] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:10:51 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600578/ [7600579] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:30:39 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600579/ [7600580] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:18:21 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600580/ [7600581] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600581/ [7600582] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:41:55 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600582/ [7600583] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:41:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600583/ [7600584] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600584/ [7600585] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:30:34 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600585/ [7600586] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:15:49 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600586/ [7600587] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600587/ [7600588] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:26:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600588/ [7600591] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} time: 01:13:45 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600591/ [7600592] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600592/ [7600593] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_latest}} time: 00:38:32 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600593/ [7600594] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:09:50 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600594/ [7600596] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:34 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600596/ [7600598] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:20:47 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600598/ [7600600] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600600/ [7600601] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:49 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600601/ [7600602] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:07:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600602/ [7600603] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:10:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600603/ [7600604] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:10 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600604/ [7600606] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:47 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600606/ [7600607] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} time: 00:35:12 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600607/ [7600608] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600608/ [7600609] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:21:34 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600609/ [7600610] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:16:17 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600610/ [7600611] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:12 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600611/ [7600612] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:17 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600612/ [7600613] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600613/ [7600614] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:43 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600614/ [7600615] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600615/ [7600616] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:21:34 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600616/ [7600618] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:15 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600618/ [7600619] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:27:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600619/ [7600620] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:21:49 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600620/ [7600621] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:13:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600621/ [7600622] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600622/ [7600623] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:15:18 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600623/ [7600624] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:19:50 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600624/ [7600625] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600625/ [7600626] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600626/ [7600627] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:06:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600627/ [7600628] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:23:24 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600628/ [7600631] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/rados_python} time: 00:14:07 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600631/ [7600632] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:13:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600632/ [7600633] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600633/ [7600634] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:55 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600634/ [7600635] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:31:43 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600635/ [7600636] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:15:59 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600636/ [7600637] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:09:47 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600637/ [7600638] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:17:36 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600638/ [7600640] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600640/ [7600641] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:15 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600641/ [7600642] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:24:48 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600642/ [7600643] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:29:43 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600643/ [7600644] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:14:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600644/ [7600645] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:13:55 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600645/ [7600646] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:29:09 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600646/ [7600647] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:51 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600647/ [7600648] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:45 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600648/ [7600649] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600649/ [7600650] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:19:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600650/ [7600651] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:29:35 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600651/ [7600652] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:19:02 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600652/ [7600655] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} time: 01:36:26 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600655/ [7600656] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600656/ [7600657] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:23 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600657/ [7600658] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:33:30 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600658/ [7600659] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:18:54 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600659/ [7600660] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:18:58 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600660/ [7600661] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:21:09 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600661/ [7600662] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600662/ [7600665] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600665/ [7600666] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:11 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600666/ [7600667] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:15:13 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600667/ [7600668] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600668/ [7600670] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600670/ [7600672] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:29 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600672/ [7600673] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:08:28 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600673/ [7600674] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:53 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600674/ [7600675] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:09:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600675/ [7600676] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:12 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600676/ [7600678] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:24:54 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600678/ [7600679] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:21:47 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600679/ [7600680] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} time: 00:11:50 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600680/ [7600682] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:17:53 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600682/ [7600683] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:18:22 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600683/ [7600684] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:24:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600684/ [7600686] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:12:08 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600686/ [7600687] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:19 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600687/ [7600688] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600688/ [7600690] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:16:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600690/ [7600691] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:22:52 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600691/ [7600695] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600695/ [7600696] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:15 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600696/ [7600697] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:37 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600697/ [7600699] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:08:12 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600699/ [7600702] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:21:37 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600702/ [7600703] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:40:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600703/ [7600704] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache} time: 00:10:51 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600704/ [7600705] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats} time: 00:09:26 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600705/ [7600706] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:38 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600706/ [7600707] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:19:38 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600707/ [7600708] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:48 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600708/ [7600709] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:16:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600709/ [7600710] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} time: 00:13:21 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600710/ [7600711] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:59 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600711/ [7600712] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:12:54 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600712/ [7600713] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:09:52 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600713/ [7600714] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:24:25 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600714/ [7600715] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} time: 00:11:25 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600715/ [7600716] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:21:28 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600716/ [7600717] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600717/ [7600718] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:26 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600718/ [7600719] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} time: 00:19:15 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600719/ [7600720] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:37:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600720/ [7600721] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:15:03 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600721/ [7600724] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:25:09 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600724/ [7600725] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600725/ [7600726] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:27:11 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600726/ [7600727] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600727/ [7600728] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:23 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600728/ [7600730] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:30:31 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600730/ [7600731] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/progress} time: 00:20:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600731/ [7600732] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600732/ [7600733] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:07 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600733/ [7600734] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 01:02:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600734/ [7600735] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:53:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600735/ [7600737] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:23:31 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600737/ [7600738] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:52 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600738/ [7600739] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:17:15 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600739/ [7600740] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600740/ [7600741] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:25:36 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600741/ [7600742] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} time: 01:13:52 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600742/ [7600744] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} time: 00:24:13 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600744/ [7600746] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:26:21 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600746/ [7600747] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600747/ [7600748] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:37 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600748/ [7600750] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:19:14 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600750/ [7600751] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:51 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600751/ [7600752] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:25:28 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600752/ [7600754] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:10:59 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600754/ [7600755] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:10:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600755/ [7600756] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600756/ [7600757] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:18 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600757/ [7600759] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:11:32 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600759/ [7600760] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 02:29:33 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600760/ [7600762] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600762/ [7600763] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:33:05 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600763/ [7600764] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:14:37 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600764/ [7600765] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:23 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600765/ [7600766] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600766/ [7600767] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:18:12 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600767/ [7600768] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:32:02 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600768/ [7600769] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} time: 00:30:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600769/ [7600770] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} time: 01:02:10 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600770/ [7600772] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600772/ [7600773] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:04 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600773/ [7600774] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:32:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600774/ [7600777] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:22:20 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600777/ [7600778] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:27:44 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600778/ [7600779] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:23 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600779/ [7600780] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:23:09 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600780/ [7600781] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:27 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600781/ [7600782] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:46:31 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600782/ [7600783] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:22:01 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600783/ [7600785] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/repair_test} time: 00:17:26 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600785/ [7600786] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:31:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600786/ [7600787] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:18 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600787/ [7600788] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:32 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600788/ [7600789] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:58 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600789/ [7600790] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600790/ [7600791] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:30:42 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600791/ [7600792] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:26:14 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600792/ [7600793] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:10:22 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600793/ [7600794] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:40 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600794/ [7600795] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:45 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600795/ [7600797] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:09:41 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600797/ [7600798] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:22 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600798/ [7600799] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:33:24 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600799/ [7600800] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:34:32 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600800/ [7600802] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600802/ [7600803] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:54 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600803/ [7600804] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:48 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600804/ [7600805] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600805/ [7600807] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:19:00 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600807/ [7600808] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:04 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600808/ [7600809] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:17 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600809/ [7600810] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:55 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600810/ [7600812] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:53 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600812/ [7600813] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600813/ [7600814] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600814/ [7600816] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:19:11 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600816/ [7600818] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/crash} time: 00:12:24 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600818/ [7600820] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:23:53 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600820/ [7600821] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:56 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600821/ [7600823] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:12:56 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600823/ [7600824] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:25 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600824/ [7600828] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:23 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600828/ [7600829] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:12:02 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600829/ [7600830] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:30:16 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600830/ [7600831] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:16:19 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600831/ [7600832] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:11:07 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600832/ [7600834] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} time: 00:10:06 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600834/ [7600835] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:30 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600835/ [7600836] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:09:38 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600836/ [7600838] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:37 info: https://pulpito.ceph.com/yuriw-2024-03-14_15:48:00-rados-wip-yuri2-testing-2024-03-13-0827-distro-default-smithi/7600838/