2024-04-04T17:54:30.048 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-04T17:54:30.048 INFO:teuthology.results:subset: '111/120000' 2024-04-04T17:54:30.048 INFO:teuthology.results:seed: '3679' 2024-04-04T20:43:22.997 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-04T20:43:23.027 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-05T08:04:28.978 INFO:teuthology.results:Tests finished! gathering results... 2024-04-05T08:04:52.063 INFO:scrape:Found 294 jobs 2024-04-05T08:04:53.288 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640133/teuthology.log 2024-04-05T08:05:12.959 INFO:scrape:Found 15 distinct failure reasons 2024-04-05T08:05:12.960 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-04-05T08:05:12.960 INFO:scrape:42 jobs 2024-04-05T08:05:12.961 INFO:scrape:suites intersection: ['rados'] 2024-04-05T08:05:12.961 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/3-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-balancer/read', 'd-balancer/upmap-read', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/monthrash/{ceph', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/readwrite}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/dedup-io-mixed}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] 2024-04-05T08:05:12.961 INFO:scrape: 2024-04-05T08:05:12.961 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-05T08:05:12.961 INFO:scrape:6 jobs: ['7639870', '7639959', '7639800', '7640121', '7640054', '7639961'] 2024-04-05T08:05:12.962 INFO:scrape:suites intersection: [] 2024-04-05T08:05:12.962 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-04-05T08:05:12.962 INFO:scrape: 2024-04-05T08:05:12.962 INFO:scrape:Dead: None 2024-04-05T08:05:12.962 INFO:scrape:3 jobs: ['7640133', '7639829', '7639787'] 2024-04-05T08:05:12.962 INFO:scrape: 2024-04-05T08:05:12.962 INFO:scrape:Failure: SSH connection to smithi177 was lost: 'sudo yum install -y kernel' 2024-04-05T08:05:12.962 INFO:scrape:1 jobs: ['7639830'] 2024-04-05T08:05:12.963 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc'] 2024-04-05T08:05:12.963 INFO:scrape: 2024-04-05T08:05:12.963 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-04-05T08:05:12.963 INFO:scrape:2 jobs: ['7640019', '7639810'] 2024-04-05T08:05:12.963 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-5}', 'ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-04-05T08:05:12.963 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/read', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-04-05T08:05:12.963 INFO:scrape: 2024-04-05T08:05:12.963 INFO:scrape:Failure: failed to complete snap trimming before timeout 2024-04-05T08:05:12.963 INFO:scrape:2 jobs: ['7639837', '7640041'] 2024-04-05T08:05:12.964 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-04-05T08:05:12.964 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-04-05T08:05:12.964 INFO:scrape: 2024-04-05T08:05:12.964 INFO:scrape:Dead: b'2024-04-05T06:39:39.237 DEBUG:teuthology.exit:Finished running handlers' 2024-04-05T08:05:12.964 INFO:scrape:2 jobs: ['7639835', '7640039'] 2024-04-05T08:05:12.964 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-05T08:05:12.964 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-05T08:05:12.964 INFO:scrape: 2024-04-05T08:05:12.964 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-04-05T08:05:12.964 INFO:scrape:4 jobs: ['7640123', '7640075', '7639874', '7639941'] 2024-04-05T08:05:12.965 INFO:scrape:suites intersection: ['rados'] 2024-04-05T08:05:12.965 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-snapmapper', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-04-05T08:05:12.965 INFO:scrape: 2024-04-05T08:05:12.965 INFO:scrape:Failure: reached maximum tries (801) after waiting for 4800 seconds 2024-04-05T08:05:12.965 INFO:scrape:1 jobs: ['7639946'] 2024-04-05T08:05:12.965 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-radosbench}'] 2024-04-05T08:05:12.965 INFO:scrape: 2024-04-05T08:05:12.965 INFO:scrape:Failure: Command failed on smithi062 with status 5: 'sudo systemctl stop ceph-5d43e03e-f2b2-11ee-b649-cb9ed24678a4@mon.a' 2024-04-05T08:05:12.965 INFO:scrape:1 jobs: ['7639806'] 2024-04-05T08:05:12.966 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-04-05T08:05:12.966 INFO:scrape: 2024-04-05T08:05:12.966 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-04-05T08:05:12.966 INFO:scrape:6 jobs: ['7640082', '7639856', '7640076', '7639868', '7640072', '7639860'] 2024-04-05T08:05:12.966 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2024-04-05T08:05:12.966 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-balancer/read', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] 2024-04-05T08:05:12.966 INFO:scrape: 2024-04-05T08:05:12.966 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-05T08:05:12.967 INFO:scrape:1 jobs: ['7639817'] 2024-04-05T08:05:12.967 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] 2024-04-05T08:05:12.967 INFO:scrape: 2024-04-05T08:05:12.967 INFO:scrape:Failure: "2024-04-04T19:40:00.000236+0000 mon.a (mon.0) 953 : cluster 3 [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm" in cluster log 2024-04-05T08:05:12.967 INFO:scrape:1 jobs: ['7639935'] 2024-04-05T08:05:12.967 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_host_drain}'] 2024-04-05T08:05:12.967 INFO:scrape: 2024-04-05T08:05:12.967 INFO:scrape:Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6094 (apt-get)'] 2024-04-05T08:05:12.967 INFO:scrape:1 jobs: ['7640029'] 2024-04-05T08:05:12.968 INFO:scrape:suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'workloads/rados_mon_osdmap_prune}'] 2024-04-05T08:05:12.968 INFO:scrape: 2024-04-05T08:05:12.968 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-04-05T08:05:12.968 INFO:scrape:1 jobs: ['7639805'] 2024-04-05T08:05:12.968 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}'] 2024-04-05T08:05:12.968 INFO:scrape: 2024-04-05T08:05:12.979 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/ failed: 67 dead: 7 running: 0 waiting: 0 queued: 0 passed: 220 Fail ================================================================= [7639785] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:12:53 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639785/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639785/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7639789] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:08:49 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639789/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639789/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7639796] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:11:23 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639796/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639796/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7639800] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:47:05 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639800/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639800/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7639802] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:08:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639802/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639802/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639805] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:52:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639805/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639805/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d863afa0d0cb46c98b86e12e5b58fc3f Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7639806] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639806/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df29002336a6443ca606f4937352022d Command failed on smithi062 with status 5: 'sudo systemctl stop ceph-5d43e03e-f2b2-11ee-b649-cb9ed24678a4@mon.a' [7639809] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:11:10 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639809/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639809/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --pool- snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639810] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:43:24 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0953358910a44b18e9fb34b3b0ee8d1 Command failed (workunit test rados/test.sh) on smithi046 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7639817] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:12:36 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0eda4e9c50634e5485a9c14658c874fc Command failed (workunit test cephadm/test_cephadm.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7639826] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:08:59 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639826/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639826/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639837] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:52:07 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639837/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=834d5c2a87644750b950e9316dfaccb3 failed to complete snap trimming before timeout [7639840] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:11:37 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639840/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639840/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 50 --op write 50 --op delete 10 --op tier_promote 30 --op write_excl 50 --pool unique_pool_0' [7639843] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:11:09 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639843/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639843/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op copy_from 100 --op write_excl 50 --pool unique_pool_0' [7639847] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:09:05 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639847/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639847/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639850] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639850/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639850/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 4000 --objects 300 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op tier_promote 10 --op write_excl 50 --pool unique_pool_0' [7639854] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:11:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639854/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639854/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7639856] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:10:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639856/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639856/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639860] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:12:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639860/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639860/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639868] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:09:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639868/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639868/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639870] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:07:38 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639870/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639870/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7639874] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:43 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639874/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639874/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bcf306ee3a3419da02690ee249b3346 Command failed on smithi119 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_snap_mapper' [7639875] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:08:52 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639875/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639875/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639880] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:08:10 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639880/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639880/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639887] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:08:47 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639887/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639887/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639899] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:09:43 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639899/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639899/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7639900] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:11:23 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639900/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise- dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' [7639920] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:14:07 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639920/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639920/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7639925] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:10:07 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639925/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639925/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7639935] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:15:49 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639935/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639935/ "2024-04-04T19:40:00.000236+0000 mon.a (mon.0) 953 : cluster 3 [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm" in cluster log [7639938] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:08:24 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639938/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639938/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7639941] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639941/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=610b06e2968342c096cc36c31f18b2b9 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7639946] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 02:50:08 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639946/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639946/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4c6fdd8a40e4330b387c73858334032 reached maximum tries (801) after waiting for 4800 seconds [7639948] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:11:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639948/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639948/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7639958] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:09:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639958/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639958/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7639959] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:18:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639959/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639959/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7639961] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:29:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639961/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639961/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7639963] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:09:36 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639963/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639963/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --pool- snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7639969] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:13:03 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639969/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639969/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7639979] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:09:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639979/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639979/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7639989] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:11:49 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639989/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639989/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7639997] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:08:21 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639997/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639997/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7640008] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:09:06 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640008/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640008/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640019] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:14:45 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640019/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640019/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0a4afc5ea614fa08ffc44a717a88600 Command failed (workunit test rados/test.sh) on smithi086 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=66deef1598934c3b1ef2f044e380a7d00244cead TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7640026] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:12:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640026/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640026/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7640041] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:40:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640041/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640041/ sentry: https://sentry.ceph.com/organizations/ceph/?query=246905c273984e02a20240cfe132848d failed to complete snap trimming before timeout [7640046] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:16:08 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640046/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640046/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640051] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:08:55 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640051/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640051/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 50 --op write 50 --op delete 10 --op tier_promote 30 --op write_excl 50 --pool unique_pool_0' [7640054] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:39:21 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640054/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640054/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7640058] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:12:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640058/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640058/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op copy_from 100 --op write_excl 50 --pool unique_pool_0' [7640060] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/readwrite} ----------------------------------------------------------------- time: 00:08:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640060/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640060/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op delete 10 --op write_excl 22 --pool unique_pool_0' [7640063] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:09:04 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640063/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640063/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 4000 --objects 300 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op tier_promote 10 --op write_excl 50 --pool unique_pool_0' [7640072] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:12:37 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640072/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640072/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640074] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640074/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640074/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7640075] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640075/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3cd8857b9ea4fb39a4ad8bc317cbdb8 Command failed on smithi181 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_snap_mapper' [7640076] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:11:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640076/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640076/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640079] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640079/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640079/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7640082] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:08:56 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640082/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640082/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640090] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:08:36 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640090/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640090/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640094] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:09:18 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640094/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640094/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640102] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:11:11 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640102/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640102/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640109] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:09:21 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640109/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640109/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise- dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' [7640115] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:11:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640115/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640115/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7640121] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:06:07 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640121/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7640122] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:30:06 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640122/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640122/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7640123] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:07:37 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640123/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640123/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7174d250b5f74784b5d5e916f8d2544f Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7640128] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:11:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640128/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640128/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' Dead ================================================================= [7639787] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639787/ Error reimaging machines: Failed to power on smithi173 [7639829] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639829/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639829/ Error reimaging machines: Expected smithi177's OS to be centos 8 but found centos 9 [7639830] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639830/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7bf7c7affbad47bf883f38d2955cba0c SSH connection to smithi177 was lost: 'sudo yum install -y kernel' [7639835] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639835/ hit max job timeout [7640029] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:07:04 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640029/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640029/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db0d8645163741408269dd8fee70fe2b ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6094 (apt-get)'] [7640039] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640039/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640039/ hit max job timeout [7640133] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640133/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640133/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7639786] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639786/ [7639788] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:14:24 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639788/ [7639790] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639790/ [7639791] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:09:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639791/ [7639792] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:19:06 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639792/ [7639793] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:23:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639793/ [7639794] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639794/ [7639795] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:53 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639795/ [7639797] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:09 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639797/ [7639798] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:21:01 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639798/ [7639799] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:44 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639799/ [7639801] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639801/ [7639803] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:09 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639803/ [7639804] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639804/ [7639807] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639807/ [7639808] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:40:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639808/ [7639811] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639811/ [7639812] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_workunits} time: 00:29:02 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639812/ [7639813] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:07:37 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639813/ [7639814] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:46:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639814/ [7639815] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639815/ [7639816] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:24:56 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639816/ [7639818] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:08 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639818/ [7639819] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:35:52 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639819/ [7639820] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:19:41 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639820/ [7639821] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639821/ [7639822] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:28:51 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639822/ [7639823] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:41:47 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639823/ [7639824] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:21 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639824/ [7639825] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:30:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639825/ [7639827] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639827/ [7639828] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:28:51 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639828/ [7639831] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:08:55 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639831/ [7639832] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/crash} time: 00:12:18 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639832/ [7639833] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:40:10 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639833/ [7639834] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:09:35 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639834/ [7639836] rados/standalone/{supported-random-distro$/{centos_latest} workloads/c2c} time: 00:14:32 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639836/ [7639838] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:15:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639838/ [7639839] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:12:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639839/ [7639841] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639841/ [7639842] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:07:51 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639842/ [7639844] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639844/ [7639845] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639845/ [7639846] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639846/ [7639848] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639848/ [7639849] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:21:27 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639849/ [7639851] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:10:35 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639851/ [7639852] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:15:13 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639852/ [7639853] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:36 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639853/ [7639855] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:12:33 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639855/ [7639857] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:31 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639857/ [7639858] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:13 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639858/ [7639859] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:30:06 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639859/ [7639861] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:10:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639861/ [7639862] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:38 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639862/ [7639863] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_latest} tasks/failover} time: 00:13:33 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639863/ [7639864] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:19:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639864/ [7639865] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639865/ [7639866] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:01 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639866/ [7639867] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:06:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639867/ [7639869] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:31:43 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639869/ [7639871] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:17:08 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639871/ [7639872] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:10 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639872/ [7639873] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:17 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639873/ [7639876] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:13:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639876/ [7639877] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_latest}} time: 00:07:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639877/ [7639878] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:19:03 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639878/ [7639879] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:19:32 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639879/ [7639881] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:47 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639881/ [7639882] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:25:32 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639882/ [7639883] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:26 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639883/ [7639884] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:13:33 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639884/ [7639886] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639886/ [7639889] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:04 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639889/ [7639890] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/insights} time: 00:11:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639890/ [7639892] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:20:47 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639892/ [7639894] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:19:17 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639894/ [7639895] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:23:03 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639895/ [7639897] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:14:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639897/ [7639902] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:36:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639902/ [7639903] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:05 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639903/ [7639905] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:13 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639905/ [7639907] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:32:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639907/ [7639908] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:29:31 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639908/ [7639910] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:19:57 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639910/ [7639911] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:17:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639911/ [7639913] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639913/ [7639915] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639915/ [7639916] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:25:09 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639916/ [7639918] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:09 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639918/ [7639919] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639919/ [7639921] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:14:32 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639921/ [7639922] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} tasks/e2e} time: 00:23:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639922/ [7639923] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:43 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639923/ [7639927] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:26 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639927/ [7639928] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:07:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639928/ [7639930] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:08:16 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639930/ [7639932] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639932/ [7639933] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:17 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639933/ [7639936] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:23:53 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639936/ [7639940] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} time: 00:11:48 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639940/ [7639943] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:16:57 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639943/ [7639945] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:16:05 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639945/ [7639949] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:11:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639949/ [7639951] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639951/ [7639953] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:15 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639953/ [7639954] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:22 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639954/ [7639956] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:16:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639956/ [7639964] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:49 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639964/ [7639966] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:11 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639966/ [7639967] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:06 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639967/ [7639971] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:10:48 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639971/ [7639972] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:10 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639972/ [7639974] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:35:21 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639974/ [7639976] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:38 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639976/ [7639977] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:37:46 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639977/ [7639980] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:11:05 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639980/ [7639982] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:41 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639982/ [7639984] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:20:13 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639984/ [7639985] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:59 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639985/ [7639987] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639987/ [7639990] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639990/ [7639992] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639992/ [7639993] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:13:10 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639993/ [7639995] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:32:23 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639995/ [7639998] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:22:26 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7639998/ [7640000] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:51 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640000/ [7640002] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:33 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640002/ [7640003] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:23:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640003/ [7640005] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:19 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640005/ [7640006] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:12:20 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640006/ [7640010] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:26:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640010/ [7640011] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:25:59 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640011/ [7640013] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:17 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640013/ [7640015] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:25:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640015/ [7640016] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640016/ [7640018] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:15:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640018/ [7640021] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:33:53 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640021/ [7640023] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:23:13 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640023/ [7640024] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640024/ [7640028] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 01:03:34 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640028/ [7640031] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:44 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640031/ [7640033] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:25:49 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640033/ [7640034] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:07 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640034/ [7640035] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:16:23 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640035/ [7640036] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:16 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640036/ [7640037] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:24:32 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640037/ [7640038] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:09:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640038/ [7640040] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} time: 00:23:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640040/ [7640042] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:00:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640042/ [7640043] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:11:55 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640043/ [7640044] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:24 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640044/ [7640045] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:13:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640045/ [7640047] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:24 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640047/ [7640048] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:19:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640048/ [7640049] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:26 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640049/ [7640050] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:10:56 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640050/ [7640052] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:35 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640052/ [7640053] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640053/ [7640055] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:10:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640055/ [7640056] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 02:39:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640056/ [7640057] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640057/ [7640059] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} time: 02:08:38 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640059/ [7640061] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:11 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640061/ [7640062] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640062/ [7640064] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:47:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640064/ [7640065] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} time: 00:28:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640065/ [7640066] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:07:52 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640066/ [7640067] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:12:42 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640067/ [7640068] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:41 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640068/ [7640069] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:35 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640069/ [7640070] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640070/ [7640071] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:34:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640071/ [7640073] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:22:14 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640073/ [7640077] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:40 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640077/ [7640078] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:43:19 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640078/ [7640080] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:15:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640080/ [7640081] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:18:53 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640081/ [7640083] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:25 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640083/ [7640084] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:09:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640084/ [7640085] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640085/ [7640086] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640086/ [7640087] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:16:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640087/ [7640088] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:12:17 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640088/ [7640089] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:08:03 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640089/ [7640091] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:51 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640091/ [7640092] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:13 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640092/ [7640093] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:07:46 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640093/ [7640095] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:43:52 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640095/ [7640096] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:33:17 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640096/ [7640097] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:18:45 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640097/ [7640098] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:45 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640098/ [7640099] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_latest}} time: 00:06:34 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640099/ [7640100] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:07:47 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640100/ [7640101] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:59 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640101/ [7640103] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:21:46 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640103/ [7640104] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:06 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640104/ [7640105] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:54 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640105/ [7640106] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:35 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640106/ [7640107] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:21:08 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640107/ [7640108] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:48 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640108/ [7640110] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:30 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640110/ [7640111] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:58 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640111/ [7640112] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:19:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640112/ [7640113] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} time: 00:26:00 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640113/ [7640114] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640114/ [7640116] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:19:45 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640116/ [7640117] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640117/ [7640118] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640118/ [7640119] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:12:28 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640119/ [7640120] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:34 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640120/ [7640124] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:12:56 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640124/ [7640125] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:09:31 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640125/ [7640126] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:29:27 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640126/ [7640127] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:12:12 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640127/ [7640129] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:16:29 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640129/ [7640130] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:38 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640130/ [7640131] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:49 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640131/ [7640132] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:09:35 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640132/ [7640134] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:11:50 info: https://pulpito.ceph.com/yuriw-2024-04-04_14:50:58-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7640134/