2024-03-30T10:24:04.196 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-30T10:24:04.196 INFO:teuthology.results:subset: '111/120000' 2024-03-30T10:24:04.196 INFO:teuthology.results:seed: '9805' 2024-03-30T12:47:01.179 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-30T12:47:01.214 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-31T00:09:04.223 INFO:teuthology.results:Tests finished! gathering results... 2024-03-31T00:09:29.169 INFO:scrape:Found 294 jobs 2024-03-31T00:09:29.697 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f3a6eb81517] , 2: ceph_test_rados(+0x28231) [0x55b4072a7231] 2024-03-31T00:09:29.697 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:29.743 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f3a6eb81517] , 2: ceph_test_rados(+0x28231) [0x55b4072a7231] 2024-03-31T00:09:29.744 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:29.786 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f3a6eb81517] , 2: ceph_test_rados(+0x28231) [0x55b4072a7231] 2024-03-31T00:09:29.786 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:29.828 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f3a6eb81517] , 2: ceph_test_rados(+0x28231) [0x55b4072a7231] 2024-03-31T00:09:29.828 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:34.031 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f19d41b63f8] , 2: ceph_test_rados(+0x28b2f) [0x55a0317f8b2f] 2024-03-31T00:09:34.031 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:34.047 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f19d41b63f8] , 2: ceph_test_rados(+0x28b2f) [0x55a0317f8b2f] 2024-03-31T00:09:34.047 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:37.531 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630678/teuthology.log 2024-03-31T00:09:38.271 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc86cc483f8] , 2: ceph_test_rados(+0x28b2f) [0x55b713bc3b2f] 2024-03-31T00:09:38.272 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:38.296 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc86cc483f8] , 2: ceph_test_rados(+0x28b2f) [0x55b713bc3b2f] 2024-03-31T00:09:38.297 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:39.619 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7ff69f581517] , 2: ceph_test_rados(+0x28231) [0x557f6a1a9231] 2024-03-31T00:09:39.620 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:39.951 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7812981517] , 2: ceph_test_rados(+0x28231) [0x55a45fbb8231] 2024-03-31T00:09:39.951 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:41.025 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc81a781517] , 2: ceph_test_rados(+0x28231) [0x563ae5314231] 2024-03-31T00:09:41.026 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:41.065 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc81a781517] , 2: ceph_test_rados(+0x28231) [0x563ae5314231] 2024-03-31T00:09:41.065 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:53.104 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fce16781517] , 2: ceph_test_rados(+0x28231) [0x560d4ee29231] 2024-03-31T00:09:53.104 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:53.125 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fce16781517] , 2: ceph_test_rados(+0x28231) [0x560d4ee29231] 2024-03-31T00:09:53.125 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-31T00:09:53.294 INFO:scrape:Found 17 distinct failure reasons 2024-03-31T00:09:53.295 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' 2024-03-31T00:09:53.295 INFO:scrape:14 jobs: ['7630688', '7630524', '7630684', '7630562', '7630691', '7630697', '7630561', '7630528', '7630641', '7630675', '7630724', '7630522', '7630518', '7630536'] 2024-03-31T00:09:53.295 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2024-03-31T00:09:53.295 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-balancer/read', 'd-balancer/upmap-read', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-overwrites}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}'] 2024-03-31T00:09:53.295 INFO:scrape: 2024-03-31T00:09:53.295 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-03-31T00:09:53.296 INFO:scrape:32 jobs 2024-03-31T00:09:53.296 INFO:scrape:suites intersection: ['rados'] 2024-03-31T00:09:53.296 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/3-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-balancer/read', 'd-balancer/upmap-read', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/monthrash/{ceph', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/readwrite}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/dedup-io-mixed}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] 2024-03-31T00:09:53.296 INFO:scrape: 2024-03-31T00:09:53.296 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-03-31T00:09:53.297 INFO:scrape:4 jobs: ['7630738', '7630542', '7630689', '7630589'] 2024-03-31T00:09:53.297 INFO:scrape:suites intersection: ['rados'] 2024-03-31T00:09:53.297 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-snapmapper', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-31T00:09:53.297 INFO:scrape: 2024-03-31T00:09:53.297 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-03-31T00:09:53.297 INFO:scrape:1 jobs: ['7630485'] 2024-03-31T00:09:53.297 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] 2024-03-31T00:09:53.297 INFO:scrape: 2024-03-31T00:09:53.297 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2024-03-31T00:09:53.297 INFO:scrape:1 jobs: ['7630577'] 2024-03-31T00:09:53.298 INFO:scrape:suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zlib}', 'tasks/e2e}'] 2024-03-31T00:09:53.298 INFO:scrape: 2024-03-31T00:09:53.298 INFO:scrape:Failure: Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-b9925c2c-ee83-11ee-b647-cb9ed24678a4@mon.a' 2024-03-31T00:09:53.298 INFO:scrape:1 jobs: ['7630474'] 2024-03-31T00:09:53.298 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_ca_signed_key}'] 2024-03-31T00:09:53.298 INFO:scrape: 2024-03-31T00:09:53.298 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-31T00:09:53.298 INFO:scrape:6 jobs: ['7630600', '7630666', '7630468', '7630601', '7630538', '7630736'] 2024-03-31T00:09:53.298 INFO:scrape:suites intersection: [] 2024-03-31T00:09:53.299 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-03-31T00:09:53.299 INFO:scrape: 2024-03-31T00:09:53.299 INFO:scrape:Failure: No module named 'tasks.ceph' 2024-03-31T00:09:53.299 INFO:scrape:5 jobs: ['7630627', '7630612', '7630617', '7630623', '7630643'] 2024-03-31T00:09:53.299 INFO:scrape:suites intersection: ['rados'] 2024-03-31T00:09:53.299 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/monthrash/{ceph', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_workunit_loadgen_mix}', 'thrashers/careful', 'thrashers/many', 'thrashers/mapgap', 'thrashers/none', 'thrashosds-health', 'workloads/cache}', 'workloads/dedup-io-snaps}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/rados_mon_osdmap_prune}'] 2024-03-31T00:09:53.299 INFO:scrape: 2024-03-31T00:09:53.299 INFO:scrape:Dead: b'2024-03-31T00:05:00.215 DEBUG:teuthology.exit:Finished running handlers' 2024-03-31T00:09:53.299 INFO:scrape:3 jobs: ['7630651', '7630498', '7630503'] 2024-03-31T00:09:53.300 INFO:scrape:suites intersection: [] 2024-03-31T00:09:53.300 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', 'mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/singleton-nomsgr/{all/admin_socket_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-31T00:09:53.300 INFO:scrape: 2024-03-31T00:09:53.300 INFO:scrape:Failure: failed to complete snap trimming before timeout 2024-03-31T00:09:53.300 INFO:scrape:2 jobs: ['7630505', '7630653'] 2024-03-31T00:09:53.300 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-03-31T00:09:53.300 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-03-31T00:09:53.300 INFO:scrape: 2024-03-31T00:09:53.300 INFO:scrape:Dead: None 2024-03-31T00:09:53.300 INFO:scrape:2 jobs: ['7630678', '7630593'] 2024-03-31T00:09:53.301 INFO:scrape: 2024-03-31T00:09:53.301 INFO:scrape:Failure: SSH connection to smithi110 was lost: 'sudo apt-get update' 2024-03-31T00:09:53.301 INFO:scrape:1 jobs: ['7630594'] 2024-03-31T00:09:53.301 INFO:scrape:suites: ['mon_election/classic', 'objectstore/bluestore-low-osd-mem-target', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_1Shard_16Threads', 'settings/optimized', 'ubuntu_latest', 'workloads/radosbench_4M_write}'] 2024-03-31T00:09:53.301 INFO:scrape: 2024-03-31T00:09:53.301 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-31T00:09:53.301 INFO:scrape:1 jobs: ['7630473'] 2024-03-31T00:09:53.301 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] 2024-03-31T00:09:53.301 INFO:scrape: 2024-03-31T00:09:53.301 INFO:scrape:Failure: Command failed on smithi129 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1864-g5dc2ea30-1jammy ceph-mds=19.0.0-1864-g5dc2ea30-1jammy ceph-common=19.0.0-1864-g5dc2ea30-1jammy ceph-fuse=19.0.0-1864-g5dc2ea30-1jammy ceph-test=19.0.0-1864-g5dc2ea30-1jammy radosgw=19.0.0-1864-g5dc2ea30-1jammy python-ceph=19.0.0-1864-g5dc2ea30-1jammy libcephfs1=19.0.0-1864-g5dc2ea30-1jammy libcephfs-java=19.0.0-1864-g5dc2ea30-1jammy libcephfs-jni=19.0.0-1864-g5dc2ea30-1jammy librados2=19.0.0-1864-g5dc2ea30-1jammy librbd1=19.0.0-1864-g5dc2ea30-1jammy rbd-fuse=19.0.0-1864-g5dc2ea30-1jammy' 2024-03-31T00:09:53.301 INFO:scrape:1 jobs: ['7630622'] 2024-03-31T00:09:53.302 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] 2024-03-31T00:09:53.302 INFO:scrape: 2024-03-31T00:09:53.302 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-03-31T00:09:53.302 INFO:scrape:2 jobs: ['7630478', '7630637'] 2024-03-31T00:09:53.302 INFO:scrape:suites intersection: ['3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'supported-random-distro$/{centos_latest}', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-03-31T00:09:53.302 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/read', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-03-31T00:09:53.302 INFO:scrape: 2024-03-31T00:09:53.302 INFO:scrape:Failure: Config file not found: "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/cephadm.conf". 2024-03-31T00:09:53.302 INFO:scrape:1 jobs: ['7630620'] 2024-03-31T00:09:53.303 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/repave-all}', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc'] 2024-03-31T00:09:53.303 INFO:scrape: 2024-03-31T00:09:53.303 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-31T00:09:53.303 INFO:scrape:2 jobs: ['7630702', '7630654'] 2024-03-31T00:09:53.303 INFO:scrape:suites intersection: [] 2024-03-31T00:09:53.303 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-inject-leak/osd', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/valgrind-leaks/{1-start', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench}'] 2024-03-31T00:09:53.303 INFO:scrape: 2024-03-31T00:09:53.313 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/ failed: 73 dead: 6 running: 0 waiting: 0 queued: 0 passed: 215 Fail ================================================================= [7630453] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:12:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630453/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630453/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7630457] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:09:38 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630457/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630457/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7630464] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:11:11 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630464/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630464/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7630468] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:46:07 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630468/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630468/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7630470] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:10:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630470/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630470/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630473] rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:32:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d9772a58cb2409e9ccaf8c865e0b437 Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7630474] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} ----------------------------------------------------------------- time: 00:11:22 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630474/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a683958f6ca0438b91ec1783b0a26575 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-b9925c2c-ee83-11ee-b647-cb9ed24678a4@mon.a' [7630477] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:11:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630477/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630477/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --pool- snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630478] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:35:04 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630478/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=662a5309900b477eac538be2594db4db Command failed (workunit test rados/test.sh) on smithi088 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7630485] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:12:38 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630485/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630485/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90932ffe3b1c468aaf01716f2d982d8d Command failed (workunit test cephadm/test_cephadm.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7630494] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:09:54 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630494/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630494/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630505] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:51:31 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630505/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6a6532909174703b31357a09b72573f failed to complete snap trimming before timeout [7630508] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630508/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630508/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 50 --op write 50 --op delete 10 --op tier_promote 30 --op write_excl 50 --pool unique_pool_0' [7630511] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:12:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630511/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630511/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op copy_from 100 --op write_excl 50 --pool unique_pool_0' [7630515] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:12:07 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630515/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630515/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630518] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:15:11 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630518/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630518/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 4000 --objects 300 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op tier_promote 10 --op write_excl 50 --pool unique_pool_0' [7630522] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:10:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630522/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630522/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7630524] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:11:41 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630524/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630524/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630528] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630528/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630528/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630536] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:11:35 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630536/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630536/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630538] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:05:29 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630538/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630538/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7630542] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:29 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630542/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630542/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f61a44ea153a4e16933e6f05f5169844 Command failed on smithi053 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_snap_mapper' [7630543] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:11:07 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630543/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630543/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630548] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:10:44 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630548/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630548/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630554] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:11:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630554/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630554/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630561] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:09:49 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630561/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630561/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7630562] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:10:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630562/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630562/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise- dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' [7630575] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:16:54 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630575/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630575/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7630577] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/e2e} ----------------------------------------------------------------- time: 00:20:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630577/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe0bd604b61d4ddd92e2b55164c93472 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7630579] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630579/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630579/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7630587] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:11:26 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630587/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630587/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7630589] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630589/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630589/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22516a5a106a4d38acf3c0c555b825c1 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7630599] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:09:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630599/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630599/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7630600] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:16:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630600/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630600/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7630601] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:33:18 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630601/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630601/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7630602] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:09:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630602/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630602/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --pool- snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630606] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:12:25 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630606/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630606/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [7630612] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:10:54 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eabec34a46eb443b905258072c46fecd No module named 'tasks.thrashosds' [7630617] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:11:15 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630617/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630617/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eaf128c859044dfd97cb8098153f88de No module named 'tasks.thrashosds' [7630618] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:09:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630618/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630618/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [7630620] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:06:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630620/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630620/ sentry: https://sentry.ceph.com/organizations/ceph/?query=040b996896b14a619a4e741b44384197 Config file not found: "/home/teuthworker/src/github.com_ceph_ceph- c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/cephadm.conf". [7630622] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:07:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630622/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=12fd83f828bd476781d7db00c8d5bfce Command failed on smithi129 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1864-g5dc2ea30-1jammy ceph-mds=19.0.0-1864-g5dc2ea30-1jammy ceph-common=19.0.0-1864-g5dc2ea30-1jammy ceph- fuse=19.0.0-1864-g5dc2ea30-1jammy ceph-test=19.0.0-1864-g5dc2ea30-1jammy radosgw=19.0.0-1864-g5dc2ea30-1jammy python- ceph=19.0.0-1864-g5dc2ea30-1jammy libcephfs1=19.0.0-1864-g5dc2ea30-1jammy libcephfs-java=19.0.0-1864-g5dc2ea30-1jammy libcephfs- jni=19.0.0-1864-g5dc2ea30-1jammy librados2=19.0.0-1864-g5dc2ea30-1jammy librbd1=19.0.0-1864-g5dc2ea30-1jammy rbd-fuse=19.0.0-1864-g5dc2ea30-1jammy' [7630623] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:07:18 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630623/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7291f6249d2b4db48502ff102eb9c1c7 No module named 'tasks.ceph' [7630627] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} ----------------------------------------------------------------- time: 00:06:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630627/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=251b46db55ce4b4b9b4d8e7d56f07cc2 No module named 'tasks.ceph' [7630630] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:13:30 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630630/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630630/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630637] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:19:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630637/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630637/ sentry: https://sentry.ceph.com/organizations/ceph/?query=850dd0eaa14f4979aec3223a9f420006 Command failed (workunit test rados/test.sh) on smithi033 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7630641] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:09:18 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630641/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630641/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7630643] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:10:31 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630643/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630643/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff0d079eef79460f94669a0214689011 No module named 'tasks' [7630653] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:41:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630653/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630653/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9106ba89c194425288004af05718eccf failed to complete snap trimming before timeout [7630654] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:17:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630654/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630654/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5253dd31bd7b432991b3cd307c135af0 reached maximum tries (501) after waiting for 3000 seconds [7630658] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:14:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630658/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630658/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630663] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:11:08 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630663/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630663/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 50 --op write 50 --op delete 10 --op tier_promote 30 --op write_excl 50 --pool unique_pool_0' [7630666] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:52:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630666/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630666/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7630670] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:13:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630670/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630670/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op copy_from 100 --op write_excl 50 --pool unique_pool_0' [7630672] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/readwrite} ----------------------------------------------------------------- time: 00:07:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630672/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630672/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op delete 10 --op write_excl 22 --pool unique_pool_0' [7630675] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:10:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630675/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630675/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 4000 --objects 300 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op tier_promote 10 --op write_excl 50 --pool unique_pool_0' [7630684] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:09:56 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630684/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630684/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630688] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:11:50 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630688/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630688/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' [7630689] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:50 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630689/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630689/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d33fdf2ba8d44e6382229c67b1bace1b Command failed on smithi131 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_snap_mapper' [7630691] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:16:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630691/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630691/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630694] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:13:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630694/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630694/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7630697] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:11:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630697/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630697/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630702] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} ----------------------------------------------------------------- time: 00:24:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630702/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630702/ reached maximum tries (51) after waiting for 300 seconds [7630705] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:10:31 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630705/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630705/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630709] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:11:36 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630709/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630709/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630717] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:08:49 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630717/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630717/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630724] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:12:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630724/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630724/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise- dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' [7630730] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:43:22 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [7630736] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:51:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630736/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630736/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7630737] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:29:02 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630737/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630737/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7630738] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:07:37 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630738/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41c4d8ad6d0b42a08110c49fac5f7439 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7630743] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:11:02 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630743/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630743/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7630748] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:11:29 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630748/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630748/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' Dead ================================================================= [7630498] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630498/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630498/ hit max job timeout [7630503] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630503/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630503/ hit max job timeout [7630593] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630593/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630593/ Error reimaging machines: Expected smithi110's OS to be centos 9 but found ubuntu 22.04 [7630594] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630594/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630594/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a05d0b4a3114e3c89edbd0e60c35c1c SSH connection to smithi110 was lost: 'sudo apt-get update' [7630651] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630651/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630651/ hit max job timeout [7630678] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630678/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630678/ Error reimaging machines: Failed to power on smithi184 Pass ================================================================= [7630454] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:25 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630454/ [7630455] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:29:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630455/ [7630456] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630456/ [7630458] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:58 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630458/ [7630459] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} time: 00:07:25 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630459/ [7630460] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} time: 00:19:22 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630460/ [7630461] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:23:35 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630461/ [7630462] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:14:34 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630462/ [7630463] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:51 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630463/ [7630465] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630465/ [7630466] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:16:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630466/ [7630467] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:11:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630467/ [7630469] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630469/ [7630471] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:17 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630471/ [7630472] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:05 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630472/ [7630475] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:10:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630475/ [7630476] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:40:32 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630476/ [7630479] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630479/ [7630480] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:30:50 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630480/ [7630481] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:09:41 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630481/ [7630482] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:36:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630482/ [7630483] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630483/ [7630484] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:37 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630484/ [7630486] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:27:07 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630486/ [7630487] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:35:01 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630487/ [7630488] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:23:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630488/ [7630489] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:17 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630489/ [7630490] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} time: 00:25:35 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630490/ [7630491] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:44:49 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630491/ [7630492] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630492/ [7630493] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:33:33 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630493/ [7630495] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:47 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630495/ [7630496] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:30:19 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630496/ [7630497] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:34:10 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630497/ [7630499] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/dashboard} time: 01:15:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630499/ [7630500] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:45 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630500/ [7630501] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:39:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630501/ [7630502] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:12:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630502/ [7630504] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630504/ [7630506] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:22:02 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630506/ [7630507] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:17:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630507/ [7630509] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630509/ [7630510] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:08:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630510/ [7630512] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:33 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630512/ [7630513] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:56 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630513/ [7630514] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630514/ [7630516] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:11:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630516/ [7630517] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} time: 00:15:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630517/ [7630519] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:10:22 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630519/ [7630520] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:17:45 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630520/ [7630521] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:47 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630521/ [7630523] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:15:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630523/ [7630525] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:19:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630525/ [7630526] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630526/ [7630527] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:30:26 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630527/ [7630529] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630529/ [7630530] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:05 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630530/ [7630531] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/failover} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630531/ [7630532] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:23:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630532/ [7630533] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630533/ [7630534] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:01 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630534/ [7630535] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:06:17 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630535/ [7630537] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:18:08 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630537/ [7630539] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:17:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630539/ [7630540] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:06 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630540/ [7630541] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:34 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630541/ [7630544] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:15:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630544/ [7630545] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_latest}} time: 00:07:15 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630545/ [7630546] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:16:47 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630546/ [7630547] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:19:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630547/ [7630549] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630549/ [7630550] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:22:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630550/ [7630551] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:17 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630551/ [7630552] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:15:30 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630552/ [7630553] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:06 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630553/ [7630555] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:02 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630555/ [7630556] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:41 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630556/ [7630557] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:20:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630557/ [7630558] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:19:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630558/ [7630559] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:26:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630559/ [7630560] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:19:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630560/ [7630563] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} time: 01:24:40 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630563/ [7630564] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630564/ [7630565] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:31 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630565/ [7630566] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:33:25 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630566/ [7630567] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:22:36 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630567/ [7630568] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:22:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630568/ [7630569] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:18:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630569/ [7630570] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630570/ [7630571] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630571/ [7630572] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:24:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630572/ [7630573] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630573/ [7630574] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630574/ [7630576] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:14:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630576/ [7630578] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:26 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630578/ [7630580] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630580/ [7630581] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:11:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630581/ [7630582] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:38 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630582/ [7630583] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:09:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630583/ [7630584] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:45 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630584/ [7630585] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} time: 00:19:05 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630585/ [7630586] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:23:33 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630586/ [7630588] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630588/ [7630590] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:17:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630590/ [7630591] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:21:33 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630591/ [7630592] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 01:31:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630592/ [7630595] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:28 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630595/ [7630596] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630596/ [7630597] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:16:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630597/ [7630598] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:18:33 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630598/ [7630603] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630603/ [7630604] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:19:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630604/ [7630605] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_big} time: 00:28:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630605/ [7630607] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:10:04 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630607/ [7630608] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:15:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630608/ [7630609] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:36:30 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630609/ [7630610] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:18:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630610/ [7630611] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:34:53 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630611/ [7630613] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:11:06 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630613/ [7630614] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630614/ [7630615] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:17:44 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630615/ [7630616] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:09:45 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630616/ [7630619] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:19:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630619/ [7630621] rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} time: 01:02:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630621/ [7630624] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:19:43 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630624/ [7630625] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:24 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630625/ [7630626] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630626/ [7630628] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:27:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630628/ [7630629] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630629/ [7630631] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} time: 00:25:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630631/ [7630632] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:24:25 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630632/ [7630633] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:17:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630633/ [7630634] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:05 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630634/ [7630635] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:25 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630635/ [7630636] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:15:10 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630636/ [7630638] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:33:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630638/ [7630639] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_latest} tasks/progress} time: 00:21:28 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630639/ [7630640] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630640/ [7630642] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:57:47 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630642/ [7630644] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630644/ [7630645] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:23:37 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630645/ [7630646] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:17 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630646/ [7630647] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:19:10 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630647/ [7630648] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:27:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630648/ [7630649] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:32:58 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630649/ [7630650] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} tasks/dashboard} time: 01:10:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630650/ [7630652] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:26:44 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630652/ [7630655] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:09:35 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630655/ [7630656] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630656/ [7630657] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:17:29 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630657/ [7630659] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630659/ [7630660] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:20:58 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630660/ [7630661] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:47 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630661/ [7630662] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:08:33 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630662/ [7630664] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630664/ [7630665] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:38 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630665/ [7630667] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630667/ [7630668] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 02:07:45 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630668/ [7630669] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:25:00 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630669/ [7630671] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}} time: 02:22:07 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630671/ [7630673] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:40 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630673/ [7630674] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:23 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630674/ [7630676] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:34:57 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630676/ [7630677] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} time: 00:30:40 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630677/ [7630679] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:13:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630679/ [7630680] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:09:39 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630680/ [7630681] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} time: 00:09:08 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630681/ [7630682] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:24:40 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630682/ [7630683] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:33:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630683/ [7630685] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:22:01 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630685/ [7630692] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:36 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630692/ [7630693] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:43:17 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630693/ [7630695] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:18:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630695/ [7630696] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:18:22 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630696/ [7630698] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:29 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630698/ [7630699] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/workunits} time: 00:11:21 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630699/ [7630700] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:35 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630700/ [7630701] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630701/ [7630703] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630703/ [7630704] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:10:42 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630704/ [7630706] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:52 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630706/ [7630707] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:13:35 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630707/ [7630708] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:01 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630708/ [7630710] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 02:53:10 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630710/ [7630711] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:31:11 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630711/ [7630712] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:16:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630712/ [7630713] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630713/ [7630714] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:09:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630714/ [7630715] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:09:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630715/ [7630716] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:37 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630716/ [7630718] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:22:19 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630718/ [7630719] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630719/ [7630720] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:30 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630720/ [7630721] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:12:09 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630721/ [7630722] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:14:50 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630722/ [7630723] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:32 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630723/ [7630725] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:12:36 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630725/ [7630726] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:05 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630726/ [7630727] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:22:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630727/ [7630728] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} time: 00:23:22 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630728/ [7630729] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:44 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630729/ [7630731] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:26:03 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630731/ [7630732] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630732/ [7630733] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:53 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630733/ [7630734] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:11:27 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630734/ [7630735] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630735/ [7630739] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:31 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630739/ [7630740] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:59 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630740/ [7630741] rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} time: 02:29:14 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630741/ [7630742] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:55 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630742/ [7630744] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:16 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630744/ [7630745] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} time: 00:11:12 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630745/ [7630746] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:12:13 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630746/ [7630747] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:11:52 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630747/ [7630749] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:11:56 info: https://pulpito.ceph.com/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630749/