2023-11-09T02:07:00.718 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T02:07:00.719 INFO:teuthology.results:subset: '1/150000' 2023-11-09T02:07:00.719 INFO:teuthology.results:seed: '4367' 2023-11-09T02:08:09.603 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T02:08:09.607 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T14:09:00.641 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-11-09T14:09:04.628 INFO:scrape:Found 50 jobs 2023-11-09T14:09:05.625 WARNING:scrape:Misunderstood line: 2023-11-09T14:09:05.788 INFO:scrape:Found 3 distinct failure reasons 2023-11-09T14:09:05.789 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-11-09T14:09:05.789 INFO:scrape:1 jobs: ['7452269'] 2023-11-09T14:09:05.790 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-11-09T14:09:05.790 INFO:scrape: 2023-11-09T14:09:05.790 INFO:scrape:Valgrind: mon (Leak_StillReachable) 2023-11-09T14:09:05.791 INFO:scrape:1 jobs: ['7452263'] 2023-11-09T14:09:05.791 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'validater/valgrind}'] 2023-11-09T14:09:05.792 INFO:scrape: 2023-11-09T14:09:05.792 INFO:scrape:Dead: b'2023-11-09T02:40:20.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored' 2023-11-09T14:09:05.792 INFO:scrape:1 jobs: ['7452267'] 2023-11-09T14:09:05.793 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] 2023-11-09T14:09:05.793 INFO:scrape: 2023-11-09T14:09:05.795 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/ failed: 2 dead: 0 running: 1 waiting: 0 queued: 0 passed: 47 Fail ================================================================= [7452263] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:36:22 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452263/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452263/ saw valgrind issues [7452269] rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 04:16:12 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452269/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=179b42e399564ba79041c4bc0c729d07 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi003 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' Running ================================================================= [7452267] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452267/ Pass ================================================================= [7452250] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:22:19 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452250/ [7452251] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:19:28 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452251/ [7452252] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:51 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452252/ [7452253] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:34:30 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452253/ [7452254] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:23:30 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452254/ [7452255] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:26:48 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452255/ [7452256] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:32:51 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452256/ [7452257] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_osdmap_prune} time: 01:00:58 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452257/ [7452258] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:22:28 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452258/ [7452259] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:29:45 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452259/ [7452260] rados/cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:22:36 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452260/ [7452261] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:14:20 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452261/ [7452262] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:32:00 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452262/ [7452264] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:37:19 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452264/ [7452265] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:42:38 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452265/ [7452266] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:19:29 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452266/ [7452268] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:25:25 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452268/ [7452270] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:21:54 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452270/ [7452271] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:28:52 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452271/ [7452272] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm} time: 00:22:52 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452272/ [7452273] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:12:03 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452273/ [7452274] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:16:36 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452274/ [7452275] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:40:10 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452275/ [7452276] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} time: 00:18:01 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452276/ [7452277] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:21:35 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452277/ [7452278] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:26:44 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452278/ [7452279] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:18:17 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452279/ [7452280] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:31 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452280/ [7452281] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:14:54 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452281/ [7452282] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:27:45 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452282/ [7452283] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:18:49 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452283/ [7452284] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:19:41 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452284/ [7452285] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:11:50 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452285/ [7452286] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:30:25 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452286/ [7452287] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:09 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452287/ [7452288] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:23:36 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452288/ [7452289] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:31:08 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452289/ [7452290] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:25:08 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452290/ [7452291] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:15:26 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452291/ [7452292] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:14:48 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452292/ [7452293] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:20:51 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452293/ [7452294] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:38:27 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452294/ [7452295] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:35:00 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452295/ [7452296] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:17:38 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452296/ [7452297] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:34:15 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452297/ [7452298] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} time: 00:35:35 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452298/ [7452299] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:25:12 info: https://pulpito.ceph.com/yingxin-2023-11-09_02:05:23-rados-wip-yingxin-crimson-improve-mempool4-distro-default-smithi/7452299/